|
|
598ce61229
|
Removed now wrong code in convert.py, fixed strange black magic.
|
2020-09-24 08:57:02 +02:00 |
|
|
|
8f8156fd2c
|
Adressing first pass of comments.
|
2020-09-24 08:57:02 +02:00 |
|
|
|
9d3a93db5b
|
Going back for not fuse_unk by default for BPE, but add a flag to
enable it.
|
2020-09-22 16:27:09 -04:00 |
|
|
|
033b98ce59
|
Updating convert scripts with Replace normalizer.
|
2020-09-22 08:21:38 +02:00 |
|
|
|
c59b216baa
|
Fixing convert/check scripts.
|
2020-09-22 08:21:38 +02:00 |
|
|
|
b16406c900
|
Moving StripAccents within normalizer for Albert +XLNet, but now crash
in Precompiled. offsets are wrong ?
|
2020-09-22 08:21:38 +02:00 |
|
|
|
275ee6d4c4
|
Making convert script machine agnostic.
|
2020-09-22 08:21:38 +02:00 |
|
|
|
2fd1d9cf06
|
Adding a new convert script, that will convert all python Tokenizer code
into a proper Rust Tokenizer format and check it on a file.
- Also fuse_unks by default in `tokenizers`'s BPE.
|
2020-09-22 08:21:38 +02:00 |
|
