Anthony MOI 4cb77ca64c Python - Tweak BPE constructor + add some tests

2020-04-08 14:04:26 -04:00

..

decoders.rs

Require Send for all parts of the tokenizer (#222)

2020-04-08 13:35:06 -04:00

encoding.rs

Python - Update pyo3 version

2020-04-06 21:20:16 +02:00

error.rs

Python - RustFmt

2020-02-18 15:07:34 -05:00

lib.rs

Python - Add bindings for new AddedToken options

2020-03-24 20:58:45 -04:00

models.rs

Python - Tweak BPE constructor + add some tests

2020-04-08 14:04:26 -04:00

normalizers.rs

Require Send for all parts of the tokenizer (#222)

2020-04-08 13:35:06 -04:00

pre_tokenizers.rs

Require Send for all parts of the tokenizer (#222)

2020-04-08 13:35:06 -04:00

processors.rs

Require Send for all parts of the tokenizer (#222)

2020-04-08 13:35:06 -04:00

token.rs

Python - Remove warning on unused Token

2019-12-13 15:28:48 -05:00

tokenizer.rs

Python - Update pyo3 version

2020-04-06 21:20:16 +02:00

trainers.rs

Python - Update pyo3 version

2020-04-06 21:20:16 +02:00

utils.rs

Expose Tokenizer parts

2019-12-31 22:57:47 -05:00