This website requires JavaScript.
Explore
Help
Sign In
mii
/
tokenizers
Watch
1
Star
0
Fork
0
You've already forked tokenizers
mirror of
https://github.com/mii443/tokenizers.git
synced
2025-12-18 06:19:14 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
d3d9f2c76b5d43d064fe80007ce3323cf9d0f97d
tokenizers
/
bindings
/
python
/
tests
History
Anthony MOI
d3d9f2c76b
words -> word_ids & sequences -> sequence_ids
2020-11-09 16:02:07 -05:00
..
bindings
words -> word_ids & sequences -> sequence_ids
2020-11-09 16:02:07 -05:00
documentation
typo: from_files() renamed to from_file()
2020-11-04 08:15:32 -05:00
implementations
Doc - Improve snippets testing
2020-11-02 17:07:27 -05:00
__init__.py
Python - Add some tests utils
2020-04-01 17:25:55 -04:00
test_serialization.py
Fixing a bug where long tokenizer files would be incorrectly deserialized (
#459
)
2020-10-13 18:44:24 +02:00
utils.py
Doc - Improve python and node tests
2020-11-02 17:07:27 -05:00