Arthur 914576f7ed Add more support for tiktoken based tokenizers (#1493) ...

* first commit

* update

* clippy

* lint

* clippy and lint

* fmt

* revert print

* 😈

* style

* add a test

* more fmt

* Use ignore_merges

* stub

* fix

* update

* Update tokenizers/src/models/bpe/model.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* update

* rust lint

* dob; t repeat yourself

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

2024-04-15 17:26:36 +02:00

..

node

version = "0.16.0-dev.0"

2024-04-02 09:51:14 +02:00

python

Add more support for tiktoken based tokenizers (#1493)

2024-04-15 17:26:36 +02:00