Patrick von Platen dd399d2ad0 Split Pre-Tokenizer (#542) ...

* start playing around

* make a first version

* refactor

* apply make format

* add python bindings

* add some python binding tests

* correct pre-tokenizers

* update auto-generated bindings

* lint python bindings

* add code node

* add split to docs

* refactor python binding a bit

* cargo fmt

* clippy and fmt in node

* quick updates and fixes

* Oops

* Update node typings

* Update changelog

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>

2020-11-27 17:07:03 -05:00

..

_ext

Doc - Fix entity warnings when no language tag was specified

2020-11-02 17:07:27 -05:00

_static

Doc - Update for 0.9.4

2020-11-09 16:36:04 -05:00

api

Python - Improve documentation for trainers

2020-11-23 11:52:51 -05:00

installation

Doc - Add installation steps

2020-11-02 17:07:27 -05:00

components.rst

Split Pre-Tokenizer (#542)

2020-11-27 17:07:03 -05:00

conf.py

Doc - Fix some typos

2020-11-02 17:07:27 -05:00

entities.inc

Doc - Update Decoder part of the Pipeline page

2020-11-02 17:07:27 -05:00

index.rst

Doc - Extract global entities to their own file

2020-11-02 17:07:27 -05:00

pipeline.rst

Doc - Update Decoder part of the Pipeline page

2020-11-02 17:07:27 -05:00

quicktour.rst

Generate pyi, fix tests and clippy warnings

2020-11-20 13:30:44 -05:00