Nicolas Patry 1054e243e2 Fix invalid continuing subwrd prefix. (#864) ...

* Creating failing test for invalid continuing subwrd prefix.

* Test in rust + the associated fix.

* Clippy.

* Black.

2022-01-04 14:25:35 +01:00

..

bindings

Fix invalid continuing subwrd prefix. (#864)

2022-01-04 14:25:35 +01:00

documentation

Doc - Hotfix training from iterators tutorial

2021-02-03 15:50:09 -05:00

implementations

Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762)

2021-08-12 09:42:44 -04:00

__init__.py

Python - Add some tests utils

2020-04-01 17:25:55 -04:00

test_serialization.py

Fixing a bug where long tokenizer files would be incorrectly deserialized (#459)

2020-10-13 18:44:24 +02:00

utils.py

Clippy fixes. (#846)

2021-12-15 15:55:48 +01:00