mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 13:48:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python - Fix bug in bert wordpiece example script

Browse Source
This commit is contained in:
jaymody
2020-04-18 17:50:52 -04:00
parent 670f619ab5
commit a28fd29204
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/examples/train_bert_wordpiece.py
View File

@@ -36,7 +36,7 @@ tokenizer = BertWordPieceTokenizer(
) )
# And then train # And then train
trainer = tokenizer.train( tokenizer.train(
files, files,
vocab_size=10000, vocab_size=10000,
min_frequency=2, min_frequency=2,