mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-09 14:18:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
762 Commits 1 Branch 0 Tags
2a84ef12cf877f40a2e2ef9f39b11625c62124af
Commit Graph

2 Commits

Author SHA1 Message Date
Anthony MOI
1c4593cad4 Python - Remove warning on unused Token 2019-12-13 15:28:48 -05:00
Anthony MOI
39a6d04c53 Improve Python bindings 
This is an attempt at actually exposing the same structure that we use in the Rust lib. This will allow Python to instantiate Model/PreTokenizer/... with their own arguments, combining everything without relying on parsed kwargs.
 2019-11-22 17:57:36 -05:00