Arthur Zucker 345b4eba96 updates

2023-09-01 18:41:36 +00:00

..

decoders

Add content to Strip decoder to allow decoding mid tokens. (#1199)

2023-03-24 10:14:49 +01:00

implementations

Handle when precompiled charsmap is empty (#1308)

2023-07-31 14:35:24 +02:00

models

Add unigram bytefallback (#1217)

2023-06-26 10:46:59 +02:00

normalizers

Creating normalizers.Prepend (To be used instead of Metaspace). (#1194)

2023-03-24 00:33:31 +01:00

pre_tokenizers

fix documentation regarding regex (#1264)

2023-06-07 09:41:28 +02:00

processors

Adding Sequence for PostProcessor. (#1052)

2022-08-25 14:50:06 +02:00

tools

Updating python formatting. (#1079)

2022-10-05 15:29:33 +02:00

trainers

implement a simple max_sentencepiece_length into BPE (#1228)

2023-05-16 10:08:19 +02:00

__init__.py

updates

2023-09-01 18:41:36 +00:00

__init__.pyi

fix documentation regarding regex (#1264)

2023-06-07 09:41:28 +02:00