mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 05:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,554 Commits 1 Branch 0 Tags
a5f644616b45c9251f71f8549481381dfa121650
Commit Graph

2 Commits

Author SHA1 Message Date
SaulLu
da4c7b10e4 Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762) 
* add a way to specify the unknown token in `SentencePieceUnigramTokenizer`

* add test that verify that an exception is raised for the missing unknown token

* style

* add test tokens
 2021-08-12 09:42:44 -04:00
Anthony MOI
d94fa220b6 Python - Add train_from_iterator to implementations 2021-01-07 09:02:20 -05:00