mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-06 20:58:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
75464734df8f1ddafc787ec24787f43efe206cec
tokenizers/bindings/python/py_src
History
Nicolas Patry 75464734df Adding a new normalizer that strips accents by removing combining (#416) 
* Adding a new normalizer that strips accents by removing combining

characters in unicode strings.

* Adding Node bindings

+ better normalizer impl.

* Doc comment -> Regular comment.
2020-09-17 09:49:41 +02:00
..
tokenizers
Adding a new normalizer that strips accents by removing combining (#416)
2020-09-17 09:49:41 +02:00