mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-27 02:29:38 +00:00
663644e041
The join was done after replacing bytes and building subwords, which was preventing bytes across these subwords to be merged correctly. We need to join first.
Python Bindings
# This expect the rust chain to be nightly
rustup update
rustup default nightly
# In this folder:
python3 -m venv .env
source .env/bin/activate
pip install maturin
maturin develop --release
# Then test:
pip install transformers
python example.py --file <FILE_PATH>
# or
python example.py