This website requires JavaScript.
Explore
Help
Sign In
mii
/
tokenizers
Watch
1
Star
0
Fork
0
You've already forked tokenizers
mirror of
https://github.com/mii443/tokenizers.git
synced
2025-12-03 11:18:29 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
a5f644616b45c9251f71f8549481381dfa121650
tokenizers
/
bindings
/
python
/
tests
History
Nicolas Patry
a5f644616b
Fix the error test for Python 3.10 (error message is different). (
#962
)
2022-03-23 10:35:58 +01:00
..
bindings
Fix the error test for Python 3.10 (error message is different). (
#962
)
2022-03-23 10:35:58 +01:00
documentation
Doc - Hotfix training from iterators tutorial
2021-02-03 15:50:09 -05:00
implementations
Add a way to specify the unknown token in
SentencePieceUnigramTokenizer
python implem (
#762
)
2021-08-12 09:42:44 -04:00
__init__.py
Python - Add some tests utils
2020-04-01 17:25:55 -04:00
test_serialization.py
Fixing bad deserialization following inclusion of a default for
Punctuation
. (
#884
)
2022-01-17 22:28:25 +01:00
utils.py
Clippy fixes. (
#846
)
2021-12-15 15:55:48 +01:00