This website requires JavaScript.
Explore
Help
Sign In
mii
/
tokenizers
Watch
1
Star
0
Fork
0
You've already forked tokenizers
mirror of
https://github.com/mii443/tokenizers.git
synced
2025-12-10 14:48:22 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
f94a2747029c564bbc2b07c6bbddb9dded0eec28
tokenizers
/
bindings
/
python
/
py_src
/
tokenizers
/
implementations
History
Nicolas Patry
a410903051
Upgrading to black 20.8b1
2020-09-24 09:27:30 -04:00
..
__init__.py
Failing test for compatibility for SentencePieceUnigramTokenizer.
2020-09-02 16:32:50 +02:00
base_tokenizer.py
Black pre-commit after rebase.
2020-09-24 08:57:02 +02:00
bert_wordpiece.py
New version. Staticmethods need to return a IntoPy<PyObject>
2020-09-24 08:57:02 +02:00
byte_level_bpe.py
from_files -> from_file everywhere
2020-09-24 08:57:02 +02:00
char_level_bpe.py
from_files -> from_file everywhere
2020-09-24 08:57:02 +02:00
sentencepiece_bpe.py
from_files -> from_file everywhere
2020-09-24 08:57:02 +02:00
sentencepiece_unigram.py
Upgrading to black 20.8b1
2020-09-24 09:27:30 -04:00