Nicolas Patry 2ccd16bf5c Adding a new tests for PreTokenizer.custom. ...

This example is more illustrative of what's doable for custom
PreTokenizer.

2020-10-15 10:07:48 -04:00

..

bindings

Adding a new tests for PreTokenizer.custom.

2020-10-15 10:07:48 -04:00

implementations

Upgrading to black 20.8b1

2020-09-24 09:27:30 -04:00

__init__.py

Python - Add some tests utils

2020-04-01 17:25:55 -04:00

test_serialization.py

Fixing a bug where long tokenizer files would be incorrectly deserialized (#459)

2020-10-13 18:44:24 +02:00

utils.py

Fixing a bug where long tokenizer files would be incorrectly deserialized (#459)

2020-10-13 18:44:24 +02:00