Arthur 4ea2f235b0 Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555) ...

* feature dependent test

* nit about 嗎

* update

* actuallyfix it

* update the test

add it

fix

* stub

* Update tokenizers/src/pre_tokenizers/byte_level.rs

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

* skip failing test

* add normalizer to init

---------

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

2024-07-15 12:12:03 +02:00

..

node

update dev version so 0.19.1

2024-04-17 23:17:12 +02:00

python

Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555)

2024-07-15 12:12:03 +02:00