mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 05:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
36832bfa1292b298c36447d7d2f5a3366c26644c
tokenizers/bindings/python/py_src
History
Nicolas Patry 36832bfa12 from_files -> from_file everywhere 
- read_files -> read_file
- from_file pure rust impl in python bindings
- Fix some typing in python binding
- Added {BPE,WordLevel,WordPiece}.from_file tests.
2020-09-24 08:57:02 +02:00
..
tokenizers
from_files -> from_file everywhere
2020-09-24 08:57:02 +02:00