mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-06 04:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
439305eea028b80dabcccdae9437809840968280
tokenizers/bindings/python/py_src
History
Nicolas Patry 439305eea0 Failing test for compatibility for SentencePieceUnigramTokenizer. 
- We are failing on ambiguous tokenization (AAA -> A + AA vs AA + A).
  Could be linked to float precision and hard or impossible to fix
(should not hinder model performance)

- We are now fusing_unk by default as it's the case with spm_train

- We are still failing on at least space deduplication. Probably should
  be handlded by a pre-tokenizer.
2020-09-02 16:32:50 +02:00
..
tokenizers
Failing test for compatibility for SentencePieceUnigramTokenizer.
2020-09-02 16:32:50 +02:00