mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updating the docs with the new command. (#1333)

Browse Source
This commit is contained in:
Nicolas Patry
2023-08-29 13:15:26 +02:00
committed by GitHub
parent d2010d5165
commit 8e522a38d9
5 changed files with 7 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/source-doc-builder/installation.mdx
View File

@ -52,19 +52,12 @@ cd tokenizers/bindings/python
```
At this point you should have your [virtual environment]() already
activated. In order to compile 🤗 Tokenizers, you need to install the
Python package `setuptools_rust`:
activated. In order to compile 🤗 Tokenizers, you need to:
```bash
pip install setuptools_rust
pip install -e .
```
Then you can have 🤗 Tokenizers compiled and installed in your virtual
environment with the following command:
```bash
python setup.py install
```
</python>
<rust>
## Crates.io
@ -74,7 +67,7 @@ python setup.py install
You just need to add it to your `Cargo.toml`:
```bash
tokenizers = "0.10"
cargo add tokenizers
```
</rust>
<node>
@ -86,4 +79,4 @@ You can simply install 🤗 Tokenizers with npm using:
npm install tokenizers
```
</node>
</tokenizerslangcontent>
</tokenizerslangcontent>

9
docs/source/installation/python.inc
View File

@ -37,11 +37,6 @@ Then we go into the python bindings folder::
cd tokenizers/bindings/python
At this point you should have your `virtual environment`_ already activated. In order to
compile 🤗 Tokenizers, you need to install the Python package :obj:`setuptools_rust`::
compile 🤗 Tokenizers, you need to::
pip install setuptools_rust
Then you can have 🤗 Tokenizers compiled and installed in your virtual environment with
the following command::
python setup.py install
pip install -e .