Arthur bded212356 Support None to reset pre_tokenizers and normalizers, and index sequences (#1590) ...

* initial commit

* support None

* fix clippy

* cleanup

* clean?

* propagate to pre_tokenizer

* fix test

* fix rust tests

* fix node

* propagate to decoder and post processor

* fix calls

* lint

* fmt

* node be happy I am fixing you

* initial commit

* support None

* fix clippy

* cleanup

* clean?

* propagate to pre_tokenizer

* fix test

* fix rust tests

* fix node

* propagate to decoder and post processor

* fix calls

* lint

* fmt

* node be happy I am fixing you

* add a small test

* styling

* style merge

* fix merge test

* fmt

* nits

* update tset

2024-08-07 12:52:35 +02:00

..

utils

Using serde (serde_pyo3) to get __str__ and __repr__ easily. (#1588)

2024-08-07 12:08:29 +02:00

decoders.rs

Using serde (serde_pyo3) to get __str__ and __repr__ easily. (#1588)

2024-08-07 12:08:29 +02:00

encoding.rs

PyO3 0.21. (#1494)

2024-04-16 13:49:52 +02:00

error.rs

PyO3 0.21. (#1494)

2024-04-16 13:49:52 +02:00

lib.rs

PyO3 0.21. (#1494)

2024-04-16 13:49:52 +02:00

models.rs

Using serde (serde_pyo3) to get __str__ and __repr__ easily. (#1588)

2024-08-07 12:08:29 +02:00

normalizers.rs

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00

pre_tokenizers.rs

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00

processors.rs

Using serde (serde_pyo3) to get __str__ and __repr__ easily. (#1588)

2024-08-07 12:08:29 +02:00

token.rs

pyo3: update to 0.19 (#1322)

2023-08-16 18:40:32 +02:00

tokenizer.rs

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00

trainers.rs

Using serde (serde_pyo3) to get __str__ and __repr__ easily. (#1588)

2024-08-07 12:08:29 +02:00