mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 16:49:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python - Add bert wordpiece training example

Browse Source
This commit is contained in:
Anthony MOI
2020-01-03 16:51:39 -05:00
parent 6e3efe8954
commit fab4e96b51
3 changed files with 65 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/examples/train_bytelevel_bpe.py
View File

@ -40,7 +40,7 @@ trainer = trainers.BpeTrainer.new(
vocab_size=50000,
min_frequency=2,
show_progress=True,
special_tokens=[ "<s>", "<pad>", "</s" ],
special_tokens=[ "<s>", "<pad>", "</s>" ],
initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train(trainer, files)