mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-07 13:18:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix typo in Python binding README

Browse Source 
Trailing paren causes an error
This commit is contained in:
Ivan Echevarria
2020-01-16 17:10:48 -08:00
committed by GitHub
parent 457e6c9932
commit e82722a9c2
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/README.md
View File

@@ -158,7 +158,7 @@ tokenizer = Tokenizer(models.BPE.empty())
# Customize pre-tokenization and decoding # Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel.new(add_prefix_space=True) tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel.new(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel.new()) tokenizer.decoder = decoders.ByteLevel.new()
# And then train # And then train
trainer = trainers.BpeTrainer.new(vocab_size=20000, min_frequency=2) trainer = trainers.BpeTrainer.new(vocab_size=20000, min_frequency=2)