mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update python README

Browse Source
This commit is contained in:
Anthony MOI
2019-11-24 00:55:13 -05:00
parent bafdc5e157
commit 512e85dfda
1 changed files with 6 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
bindings/python/README.md
View File

@ -14,7 +14,10 @@ maturin develop --release
# Then test: # Then test:
pip install transformers pip install transformers
python example.py --file <FILE_PATH> # Download vocab/merges from GPT-2
# or wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json
python example.py wget https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt
python examples/example.py --file <FILE_PATH> --merges gpt2-merges.txt --vocab gpt2-vocab.json
python custom_pre_tokenizer.py --merges gpt2-merges.txt --vocab gpt2-vocab.json
``` ```