Arthur 4ea2f235b0 Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555) ...

* feature dependent test

* nit about 嗎

* update

* actuallyfix it

* update the test

add it

fix

* stub

* Update tokenizers/src/pre_tokenizers/byte_level.rs

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

* skip failing test

* add normalizer to init

---------

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

2024-07-15 12:12:03 +02:00

..

bindings

Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555)

2024-07-15 12:12:03 +02:00

documentation

[remove black] And use ruff (#1436)

2024-03-12 11:24:21 +01:00

implementations

[remove black] And use ruff (#1436)

2024-03-12 11:24:21 +01:00

__init__.py

Python - Add some tests utils

2020-04-01 17:25:55 -04:00

test_serialization.py

Switch from cached_download to hf_hub_download in tests (#1547)

2024-06-11 15:26:58 +02:00

utils.py

Updating python formatting. (#1079)

2022-10-05 15:29:33 +02:00