mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-10 14:48:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,051 Commits 1 Branch 0 Tags
ece6ad91494d348bfa67d11769752df54d57a943
Commit Graph

3 Commits

Author SHA1 Message Date
Anthony MOI
90dfdc715d Expose Tokenizer parts 2019-12-31 22:57:47 -05:00
Anthony MOI
d565bbf309 Container - Add ability to execute 2019-11-26 17:51:26 -05:00
Anthony MOI
39a6d04c53 Improve Python bindings 
This is an attempt at actually exposing the same structure that we use in the Rust lib. This will allow Python to instantiate Model/PreTokenizer/... with their own arguments, combining everything without relying on parsed kwargs.
 2019-11-22 17:57:36 -05:00