Connor Boyle c2664ae13f Give error when initializing tokenizer with too high stride (#1306) ...

* Split `get_n_added_tokens` into separate method

* Modify `TokenizerImpl.with_truncation()` to raise an error if given bad parameters

* Return Python error if `tokenizer.with_truncation()` fails

* Add dummy variable assignment for `no_truncation()` case

* Unrelated fmt fix.

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

2023-07-28 09:16:44 +02:00

..

utils

fix documentation regarding regex (#1264)

2023-06-07 09:41:28 +02:00

decoders.rs

Add content to Strip decoder to allow decoding mid tokens. (#1199)

2023-03-24 10:14:49 +01:00

encoding.rs

pyo3 v0.18 migration (#1173)

2023-03-08 11:27:47 +01:00

error.rs

pyo3: update to 0.17 (#1066)

2022-10-05 16:59:01 +02:00

lib.rs

Printing warning to stderr. (#1222)

2023-04-19 14:55:24 +02:00

models.rs

Add unigram bytefallback (#1217)

2023-06-26 10:46:59 +02:00

normalizers.rs

Creating normalizers.Prepend (To be used instead of Metaspace). (#1194)

2023-03-24 00:33:31 +01:00

pre_tokenizers.rs

Fixing clippy warnings on 1.71. (#1296)

2023-07-16 15:58:38 +02:00

processors.rs

pyo3 v0.18 migration (#1173)

2023-03-08 11:27:47 +01:00

token.rs

Upgrade pyo3 to 0.16 (#956)

2022-05-05 15:48:40 +02:00

tokenizer.rs

Give error when initializing tokenizer with too high stride (#1306)

2023-07-28 09:16:44 +02:00

trainers.rs

implement a simple max_sentencepiece_length into BPE (#1228)

2023-05-16 10:08:19 +02:00