Commit Graph

5 Commits

Author SHA1 Message Date
dd399d2ad0 Split Pre-Tokenizer (#542)
* start playing around

* make a first version

* refactor

* apply make format

* add python bindings

* add some python binding tests

* correct pre-tokenizers

* update auto-generated bindings

* lint python bindings

* add code node

* add split to docs

* refactor python binding a bit

* cargo fmt

* clippy and fmt in node

* quick updates and fixes

* Oops

* Update node typings

* Update changelog

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
2020-11-27 17:07:03 -05:00
adfef0d906 Doc - Add @narsil suggestions
Co-Authored-By: Nicolas Patry <patry.nicolas@protonmail.com>
2020-11-02 17:07:27 -05:00
cdeb2c9ddb Doc - Update components page 2020-11-02 17:07:27 -05:00
12af3f2240 Doc - Quick updates and typos 2020-11-02 17:07:27 -05:00
51cd5166d9 Doc - Uniform formatting + updated content 2020-11-02 17:07:27 -05:00