Arthur bded212356 Support None to reset pre_tokenizers and normalizers, and index sequences (#1590) ...

* initial commit

* support None

* fix clippy

* cleanup

* clean?

* propagate to pre_tokenizer

* fix test

* fix rust tests

* fix node

* propagate to decoder and post processor

* fix calls

* lint

* fmt

* node be happy I am fixing you

* initial commit

* support None

* fix clippy

* cleanup

* clean?

* propagate to pre_tokenizer

* fix test

* fix rust tests

* fix node

* propagate to decoder and post processor

* fix calls

* lint

* fmt

* node be happy I am fixing you

* add a small test

* styling

* style merge

* fix merge test

* fmt

* nits

* update tset

2024-08-07 12:52:35 +02:00

..

node

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00

python

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00