mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-07 05:08:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a5f644616b45c9251f71f8549481381dfa121650
tokenizers/bindings/python/tests/implementations
History
SaulLu da4c7b10e4 Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762) 
* add a way to specify the unknown token in `SentencePieceUnigramTokenizer`

* add test that verify that an exception is raised for the missing unknown token

* style

* add test tokens
2021-08-12 09:42:44 -04:00
..
__init__.py
Python - Add first implementations tests
2020-04-01 17:25:55 -04:00
test_base_tokenizer.py
Python - Add components getter/setters to BaseTokenizer
2021-01-11 16:08:38 -05:00
test_bert_wordpiece.py
Python - Add train_from_iterator to implementations
2021-01-07 09:02:20 -05:00
test_byte_level_bpe.py
Python - Add train_from_iterator to implementations
2021-01-07 09:02:20 -05:00
test_char_bpe.py
Python - Add train_from_iterator to implementations
2021-01-07 09:02:20 -05:00
test_sentencepiece.py
Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762)
2021-08-12 09:42:44 -04:00