mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updating the docs with the new command. (#1333)

Browse Source
This commit is contained in:
Nicolas Patry
2023-08-29 13:15:26 +02:00
committed by GitHub
parent d2010d5165
commit 8e522a38d9
5 changed files with 7 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
bindings/python/README.md
View File

@ -64,8 +64,7 @@ python -m venv .env
source .env/bin/activate
# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install
pip install -e .
```
### Load a pretrained tokenizer from the Hub

31
bindings/python/build-wheels.sh
View File

@ -1,31 +0,0 @@
#!/bin/bash
set -ex
if ! command -v cargo &> /dev/null
then
curl https://sh.rustup.rs -sSf | sh -s -- -y
fi
export PATH="$HOME/.cargo/bin:$PATH"
# https://users.rust-lang.org/t/cargo-uses-too-much-memory-being-run-in-qemu/76531
echo -e "[net]\ngit-fetch-with-cli = true" > "$HOME/.cargo/config"
for PYBIN in /opt/python/cp{37,38,39,310,311}*/bin; do
export PYTHON_SYS_EXECUTABLE="$PYBIN/python"
"${PYBIN}/pip" install -U setuptools-rust setuptools wheel
"${PYBIN}/python" setup.py bdist_wheel
rm -rf build/*
done
for whl in ./dist/*.whl; do
auditwheel repair "$whl" -w dist/
done
# Keep only manylinux wheels
rm ./dist/*-linux_*
# Upload wheels
/opt/python/cp37-cp37m/bin/pip install -U awscli
/opt/python/cp37-cp37m/bin/python -m awscli s3 sync --exact-timestamps ./dist "s3://tokenizers-releases/python/$DIST_DIR"

9
bindings/python/tokenizers_manual_m1_build.sh
View File

@ -1,9 +0,0 @@
#! /bin/bash
for VARIABLE in "3.7.12" "3.8.12" "3.9.10" "3.10.2"
do
MACOSX_DEPLOYMENT_TARGET=10.11 SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" CFLAGS="-I/usr/include/openssl -I/usr/local/opt/readline/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include" CPPFLAGS="-I/usr/local/opt/zlib/include" LDFLAGS="-L/usr/lib -L/usr/local/opt/readline/lib" pyenv install $VARIABLE
~/.pyenv/versions/$VARIABLE/bin/pip install setuptools wheel setuptools-rust==0.11.3 --ignore-installed --force-reinstall
MACOSX_DEPLOYMENT_TARGET=10.11 ~/.pyenv/versions/$VARIABLE/bin/python setup.py bdist_wheel
done

15
docs/source-doc-builder/installation.mdx
View File

@ -52,19 +52,12 @@ cd tokenizers/bindings/python
```
At this point you should have your [virtual environment]() already
activated. In order to compile 🤗 Tokenizers, you need to install the
Python package `setuptools_rust`:
activated. In order to compile 🤗 Tokenizers, you need to:
```bash
pip install setuptools_rust
pip install -e .
```
Then you can have 🤗 Tokenizers compiled and installed in your virtual
environment with the following command:
```bash
python setup.py install
```
</python>
<rust>
## Crates.io
@ -74,7 +67,7 @@ python setup.py install
You just need to add it to your `Cargo.toml`:
```bash
tokenizers = "0.10"
cargo add tokenizers
```
</rust>
<node>
@ -86,4 +79,4 @@ You can simply install 🤗 Tokenizers with npm using:
npm install tokenizers
```
</node>
</tokenizerslangcontent>
</tokenizerslangcontent>

9
docs/source/installation/python.inc
View File

@ -37,11 +37,6 @@ Then we go into the python bindings folder::
cd tokenizers/bindings/python
At this point you should have your `virtual environment`_ already activated. In order to
compile 🤗 Tokenizers, you need to install the Python package :obj:`setuptools_rust`::
compile 🤗 Tokenizers, you need to::
pip install setuptools_rust
Then you can have 🤗 Tokenizers compiled and installed in your virtual environment with
the following command::
python setup.py install
pip install -e .