mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-04 03:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,554 Commits 1 Branch 0 Tags
a5f644616b45c9251f71f8549481381dfa121650
Commit Graph

2 Commits

Author SHA1 Message Date
Nicolas Patry
1a84958cc8 Fixing bad deserialization following inclusion of a default for Punctuation. (#884) 
* Fixing bad deserialization following inclusion of a default for
`Punctuation`.

* don't remove the type now...

* Adding slow test to run on all the tokenizers of the hub.

* `PartialEq` everywhere.

* Forcing `type` to exist on the `pre_tokenizers`.
 2022-01-17 22:28:25 +01:00
Nicolas Patry
88556790e7 Fixing a bug where long tokenizer files would be incorrectly deserialized (#459) 
* Fixing a bug where long tokenizer files would be incorrectly
deserialized

- Add a bunch of tests to check deserialization behaviour
- One tests also confirms current Single deserialization of Sequence.

* Better test locations for Windows + no file dependency in Python binding
Rust side.

* Adressing @n1t0 comments.
 2020-10-13 18:44:24 +02:00