Nicolas Patry 1994dcad6e Re-enabling Custom Serialize

2020-08-31 14:05:39 -04:00

..

decoders.rs

Fixes and cleanup, suggestions by @n1t0.

2020-08-04 15:59:33 -04:00

encoding.rs

Replace Model and Trainer Containers.

2020-08-04 15:59:33 -04:00

error.rs

Update Python bindings with new interface

2020-08-03 16:18:59 -04:00

lib.rs

Adding a 3 new PreTokenizers:

2020-08-31 14:05:39 -04:00

models.rs

Fixes and cleanup, suggestions by @n1t0.

2020-08-04 15:59:33 -04:00

normalizers.rs

Remove Tokenizer::normalize

2020-08-19 12:42:12 -04:00

pre_tokenizers.rs

Re-enabling Custom Serialize

2020-08-31 14:05:39 -04:00

processors.rs

Ensure serialization works in all expected ways.

2020-08-04 15:59:33 -04:00

token.rs

Python - Remove warning on unused Token

2019-12-13 15:28:48 -05:00

tokenizer.rs

Python - Improve and Test EncodeInput extraction

2020-08-21 18:39:49 -04:00

trainers.rs

Trainers don't need Arc.

2020-08-04 15:59:33 -04:00