Nicolas Patry 73637a0004 Adding ByteFallback support for tokenizers. (#1183) ...

* Adding ByteFallback support for `tokenizers`.

Two items added:

- A flag `byte_fallback` for the `BPE` model. This will be in charge
  of using `<0x61>` instead of unk on unknown tokens.
- A ByteFallback decoder, which will be in charge of putting everything
  back into string whenever possible. Showing � when the byte decoding
  fails (behavior checked against LlamaTokenizer in `transformers`.

* Update rustdoc.

* Clippy + Add BPE(byte_fallback) into bindings.

* Stupid file.

* Test artifacts removed.

* Update stub.

* Fix.

* Bad file.

* CRITICAL FIX: wrapper order because of untagged....

* Remove prints.

* Fixing <16 byte fallback.

2023-03-23 16:04:32 +01:00

..

node

Adding ByteFallback support for tokenizers. (#1183)

2023-03-23 16:04:32 +01:00

python

Adding ByteFallback support for tokenizers. (#1183)

2023-03-23 16:04:32 +01:00