This website requires JavaScript.
Explore
Help
Sign In
mii
/
tokenizers
Watch
1
Star
0
Fork
0
You've already forked tokenizers
mirror of
https://github.com/mii443/tokenizers.git
synced
2025-08-23 00:35:35 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
57200144ca2a5940a156d751ee37fe8fd1dcd85c
tokenizers
/
bindings
/
python
/
tests
History
Anthony MOI
57200144ca
Python - Fix ByteLevel instantiation from state (
#621
)
2021-02-04 10:16:05 -05:00
..
bindings
Python - Fix ByteLevel instantiation from state (
#621
)
2021-02-04 10:16:05 -05:00
documentation
Doc - Hotfix training from iterators tutorial
2021-02-03 15:50:09 -05:00
implementations
Python - Add components getter/setters to BaseTokenizer
2021-01-11 16:08:38 -05:00
__init__.py
Python - Add some tests utils
2020-04-01 17:25:55 -04:00
test_serialization.py
Fixing a bug where long tokenizer files would be incorrectly deserialized (
#459
)
2020-10-13 18:44:24 +02:00
utils.py
Doc - Improve python and node tests
2020-11-02 17:07:27 -05:00