mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-06 20:58:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,193 Commits 1 Branch 0 Tags
c0b9229833cfb3e67e2d42379f59228e24bcfe10
Commit Graph

3 Commits

Author SHA1 Message Date
Nicolas Patry
b16406c900 Moving StripAccents within normalizer for Albert +XLNet, but now crash 
in Precompiled. offsets are wrong ?
 2020-09-22 08:21:38 +02:00
Nicolas Patry
275ee6d4c4 Making convert script machine agnostic. 2020-09-22 08:21:38 +02:00
Nicolas Patry
2fd1d9cf06 Adding a new convert script, that will convert all python Tokenizer code 
into a proper Rust Tokenizer format and check it on a file.

- Also fuse_unks by default in `tokenizers`'s BPE.
 2020-09-22 08:21:38 +02:00