Connor Boyle c2664ae13f Give error when initializing tokenizer with too high stride (#1306) ...

* Split `get_n_added_tokens` into separate method

* Modify `TokenizerImpl.with_truncation()` to raise an error if given bad parameters

* Return Python error if `tokenizer.with_truncation()` fails

* Add dummy variable assignment for `no_truncation()` case

* Unrelated fmt fix.

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

2023-07-28 09:16:44 +02:00

..

node

Bump word-wrap from 1.2.3 to 1.2.4 in /bindings/node (#1299)

2023-07-21 08:08:10 +02:00

python

Give error when initializing tokenizer with too high stride (#1306)

2023-07-28 09:16:44 +02:00