mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-31 04:29:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,191 Commits 1 Branch 0 Tags
b16406c900a44e82174a4ab826a499460a2359ce
Commit Graph

3 Commits

Author SHA1 Message Date
Nicolas Patry
b16406c900 Moving StripAccents within normalizer for Albert +XLNet, but now crash 
in Precompiled. offsets are wrong ?
 2020-09-22 08:21:38 +02:00
Nicolas Patry
275ee6d4c4 Making convert script machine agnostic. 2020-09-22 08:21:38 +02:00
Nicolas Patry
2fd1d9cf06 Adding a new convert script, that will convert all python Tokenizer code 
into a proper Rust Tokenizer format and check it on a file.

- Also fuse_unks by default in `tokenizers`'s BPE.
 2020-09-22 08:21:38 +02:00