tokenizers

mirror of https://github.com/mii443/tokenizers.git synced 2025-12-07 13:18:31 +00:00

Files

Morgan Funtowicz afe9cfe96e Strip should inherits from Normalizer on Python binding.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

2020-03-31 20:20:09 +02:00

decoders.rs

2020-03-10 12:05:04 -04:00

encoding.rs

2020-03-26 15:42:45 -04:00

error.rs

Python - RustFmt

2020-02-18 15:07:34 -05:00

lib.rs

2020-03-24 20:58:45 -04:00

models.rs

2020-03-26 15:42:44 -04:00

normalizers.rs

2020-03-31 20:20:09 +02:00

pre_tokenizers.rs

2020-03-10 12:05:04 -04:00

processors.rs

2020-03-10 12:05:04 -04:00

token.rs

2019-12-13 15:28:48 -05:00

tokenizer.rs

2020-03-27 11:53:18 -04:00

trainers.rs

2020-03-24 20:58:45 -04:00

utils.rs

Expose Tokenizer parts

2019-12-31 22:57:47 -05:00