mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 05:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
da4c7b10e4766912b94ccd69dc8b91988ff86936
tokenizers/bindings/python/py_src
History
SaulLu da4c7b10e4 Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762) 
* add a way to specify the unknown token in `SentencePieceUnigramTokenizer`

* add test that verify that an exception is raised for the missing unknown token

* style

* add test tokens
2021-08-12 09:42:44 -04:00
..
tokenizers
Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762)
2021-08-12 09:42:44 -04:00