mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #80 from iechevarria/patch-1

Browse Source 
Remove trailing paren in Python binding README
This commit is contained in:
MOI Anthony
2020-01-17 06:45:32 -05:00
committed by GitHub
parent 457e6c9932 e82722a9c2
commit 5d76bf4749
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/README.md
View File

@ -158,7 +158,7 @@ tokenizer = Tokenizer(models.BPE.empty())
# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel.new(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel.new())
tokenizer.decoder = decoders.ByteLevel.new()
# And then train
trainer = trainers.BpeTrainer.new(vocab_size=20000, min_frequency=2)