This website requires JavaScript.
Explore
Help
Sign In
mii
/
tokenizers
Watch
1
Star
0
Fork
0
You've already forked tokenizers
mirror of
https://github.com/mii443/tokenizers.git
synced
2025-12-07 13:18:31 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a
tokenizers
/
bindings
/
python
/
tests
History
Nicolas Patry
2ccd16bf5c
Adding a new tests for
PreTokenizer.custom
.
...
This example is more illustrative of what's doable for custom PreTokenizer.
2020-10-15 10:07:48 -04:00
..
bindings
Adding a new tests for
PreTokenizer.custom
.
2020-10-15 10:07:48 -04:00
implementations
Upgrading to black 20.8b1
2020-09-24 09:27:30 -04:00
__init__.py
Python - Add some tests utils
2020-04-01 17:25:55 -04:00
test_serialization.py
Fixing a bug where long tokenizer files would be incorrectly deserialized (
#459
)
2020-10-13 18:44:24 +02:00
utils.py
Fixing a bug where long tokenizer files would be incorrectly deserialized (
#459
)
2020-10-13 18:44:24 +02:00