mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-09 14:18:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,097 Commits 1 Branch 0 Tags
32a76b033182b2fd211910a16f24937484e03e63
Commit Graph

3 Commits

Author SHA1 Message Date
Anthony MOI
504d8c85d8 Remove Tokenizer::normalize 
This is actually a legacy function that doesn't really make sense now, and is getting really difficult to keep. So we remove it.
 2020-08-19 12:42:12 -04:00
Anthony MOI
7833965dc4 Update Python bindings with new interface 2020-08-03 16:18:59 -04:00
Sebastian Pütz
0d7c232f95 Move Python source to subdirectory. 
This allows testing versions not built in-place. Otherwise
importing (or testing) in the package root fails without develop
builds.
Replace maturin with setuptools_rust since maturin fails with
proper project structure.
 2020-07-25 23:40:47 +02:00