mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixing missing direction in TruncationParams. (#868)

Browse Source
This commit is contained in:
Nicolas Patry
2022-01-04 14:21:46 +01:00
committed by GitHub
parent 7069988ffe
commit 4122a33f09
2 changed files with 23 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
bindings/python/test.py
View File

@ -1,12 +0,0 @@
from tokenizers import ByteLevelBPETokenizer
from tokenizers import pre_tokenizers, models, Tokenizer, trainers
tokenizer = Tokenizer(models.Unigram())
tokenizer.pre_tokenizer = pre_tokenizers.WhitespaceSplit()
trainer = trainers.UnigramTrainer(
vocab_size=400000000,
show_progress=True,
special_tokens=["<s>", "<pad>", "</s>", "<unk>", "mask"]
)
tokenizer.train(["data/big.txt"], trainer)