Nicolas Patry 0ff2ab0f64 Fixing the stream by removing the read_index altogether. (#1716) ...

* Fixing the stream by removing the read_index altogether.

* Moving the test location because.. Windows.

* Ok whatever.

* Rust 1.84

* Fmt.

2025-01-09 17:41:15 +01:00

..

common

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00

added_tokens.rs

Aho corasick version for many added tokens. (#871)

2022-01-06 16:04:51 +01:00

documentation.rs

Adding an API for decode streaming. (#1677)

2024-11-15 06:02:38 +01:00

from_pretrained.rs

Prevent using from_pretrained on invalid ids (better error message). (#1153)

2023-01-23 15:38:14 +01:00

offsets.rs

words -> word_ids & sequences -> sequence_ids

2020-11-09 16:02:07 -05:00

serialization.rs

Support None to reset pre_tokenizers and normalizers, and index sequences (#1590)

2024-08-07 12:52:35 +02:00

stream.rs

Fixing the stream by removing the read_index altogether. (#1716)

2025-01-09 17:41:15 +01:00

training.rs

Fixing clippy warnings on 1.71. (#1296)

2023-07-16 15:58:38 +02:00

unigram.rs

style: simplify string formatting for readability (#1632)

2024-10-04 13:11:50 +02:00