Nicolas Patry 2fd1d9cf06 Adding a new convert script, that will convert all python Tokenizer code ...

into a proper Rust Tokenizer format and check it on a file.

- Also fuse_unks by default in `tokenizers`'s BPE.

2020-09-22 08:21:38 +02:00

..

convert.py

Adding a new convert script, that will convert all python Tokenizer code

2020-09-22 08:21:38 +02:00

sentencepiece_extractor.py

Black pass.

2020-08-31 14:05:39 -04:00

spm_parity_check.py

Adding a new convert script, that will convert all python Tokenizer code

2020-09-22 08:21:38 +02:00