mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-10 22:58:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,349 Commits 1 Branch 0 Tags
20c7045ba17b6e79ffaa1dee34af44b61e652f81
Commit Graph

54 Commits

Author SHA1 Message Date
Anthony MOI
c036cd4ced Python - Bump version for 0.9.0.dev0 release 2020-08-21 18:52:29 -04:00
Anthony MOI
504d8c85d8 Remove Tokenizer::normalize 
This is actually a legacy function that doesn't really make sense now, and is getting really difficult to keep. So we remove it.
 2020-08-19 12:42:12 -04:00
Anthony MOI
7833965dc4 Update Python bindings with new interface 2020-08-03 16:18:59 -04:00
Sebastian Pütz
0d7c232f95 Move Python source to subdirectory. 
This allows testing versions not built in-place. Otherwise
importing (or testing) in the package root fails without develop
builds.
Replace maturin with setuptools_rust since maturin fails with
proper project structure.
 2020-07-25 23:40:47 +02:00