Arthur eea8e1ae6f Fix doc about split (#1591) ...

* update doc

* add example

* Update bindings/python/src/pre_tokenizers.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* stub

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

2024-08-07 12:35:01 +02:00

..

decoders

Fixing doc. (#1499)

2024-04-17 09:32:40 +02:00

implementations

Fixing doc. (#1499)

2024-04-17 09:32:40 +02:00

models

Fix 'dictionnary' typo (#1511)

2024-06-11 15:43:47 +02:00

normalizers

Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555)

2024-07-15 12:12:03 +02:00

pre_tokenizers

Fix doc about split (#1591)

2024-08-07 12:35:01 +02:00

processors

[remove black] And use ruff (#1436)

2024-03-12 11:24:21 +01:00

tools

[remove black] And use ruff (#1436)

2024-03-12 11:24:21 +01:00

trainers

[remove black] And use ruff (#1436)

2024-03-12 11:24:21 +01:00

__init__.py

make tests happy

2023-09-05 15:37:09 +00:00

__init__.pyi

[remove black] And use ruff (#1436)

2024-03-12 11:24:21 +01:00