dd399d2ad0
Split Pre-Tokenizer ( #542 )
...
* start playing around
* make a first version
* refactor
* apply make format
* add python bindings
* add some python binding tests
* correct pre-tokenizers
* update auto-generated bindings
* lint python bindings
* add code node
* add split to docs
* refactor python binding a bit
* cargo fmt
* clippy and fmt in node
* quick updates and fixes
* Oops
* Update node typings
* Update changelog
Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com >
2020-11-27 17:07:03 -05:00
58e1d8de67
Python - Improve documentation for trainers
2020-11-23 11:52:51 -05:00
387b8a1033
Generate pyi, fix tests and clippy warnings
2020-11-20 13:30:44 -05:00
b0d8108dcb
Doc - Update for 0.9.4
2020-11-09 16:36:04 -05:00
bc19eaa186
Doc - Fix some typos
...
Co-Authored-By: Taufiquzzaman Peyash <taufiquzzaman.peyash@northsouth.edu >
2020-11-02 17:07:27 -05:00
adfef0d906
Doc - Add @narsil suggestions
...
Co-Authored-By: Nicolas Patry <patry.nicolas@protonmail.com >
2020-11-02 17:07:27 -05:00
56c507dcdd
Doc - Basic README + last CI updates
2020-11-02 17:07:27 -05:00
48181db0d8
Doc - Fix entity warnings when no language tag was specified
2020-11-02 17:07:27 -05:00
b23310b481
Doc - Update quicktour for Node
2020-11-02 17:07:27 -05:00
9521603e08
Doc - Update Decoder part of the Pipeline page
2020-11-02 17:07:27 -05:00
8b65c1f4bc
Doc - Update Bert example on the Pipeline page
2020-11-02 17:07:27 -05:00
5839348a46
Doc - Update Model part of the Pipeline page
2020-11-02 17:07:27 -05:00
620769fd4b
Doc - Update PreTokenizer part of the Pipeline page
2020-11-02 17:07:27 -05:00
13a80050f0
Doc - Update Normalizer part of the Pipeline page
2020-11-02 17:07:27 -05:00
ab7bae466a
Doc - Better namespace for rust tests
2020-11-02 17:07:27 -05:00
cace6561d7
Doc - Extract global entities to their own file
2020-11-02 17:07:27 -05:00
816fde0d09
Doc - Update version for 0.9.3
2020-11-02 17:07:27 -05:00
290a885b76
Doc - Improve quicktour support for multi-lang
2020-11-02 17:07:27 -05:00
57391e0170
Doc - Update python versions
2020-11-02 17:07:27 -05:00
bd84240a20
Doc - Update Rust entities using roles
2020-11-02 17:07:27 -05:00
4510712295
Doc - Add some roles for Rust doc
2020-11-02 17:07:27 -05:00
519d3bd659
Doc - Add code snippets for rust in Quicktour
2020-11-02 17:07:27 -05:00
cdeb2c9ddb
Doc - Update components page
2020-11-02 17:07:27 -05:00
9dc0d73348
Doc - Entities can be anything
2020-11-02 17:07:27 -05:00
928b87eedf
Doc - Fix hf-menu overlapping
2020-11-02 17:07:27 -05:00
41bf688a49
Doc - Replace some entities in the quicktour
2020-11-02 17:07:27 -05:00
f0b6a2127c
Doc - No warning for unresolved ref + default
2020-11-02 17:07:27 -05:00
72d14d18c6
Doc - Sphinx extension to handle entities
2020-11-02 17:07:27 -05:00
4cf0a0b72c
Doc - Quicktour uses python tested code
2020-11-02 17:07:27 -05:00
108b2a6b9b
Doc - Add the GA tag
2020-11-02 17:07:27 -05:00
d2fc0e4836
Doc - Update API Reference for Encoding
2020-11-02 17:07:27 -05:00
12af3f2240
Doc - Quick updates and typos
2020-11-02 17:07:27 -05:00
403a028275
Doc - Add installation steps
2020-11-02 17:07:27 -05:00
5cb954db11
Doc - Reorganize API Reference
2020-11-02 17:07:27 -05:00
d2db0e3941
Finish the pipeline page
2020-11-02 17:07:27 -05:00
3591b3ca17
Typos + pipeline beginning
2020-11-02 17:07:27 -05:00
c4187c9369
Wrong title level
2020-11-02 17:07:27 -05:00
38aa839df0
Quicktour
2020-11-02 17:07:27 -05:00
8c0370657e
Doc - Update API Reference on more Tokenizer methods
2020-11-02 17:07:27 -05:00
ddabe130cd
Doc - Updated API Reference for AddedToken
2020-11-02 17:07:27 -05:00
79f02bb7f0
Doc - Updated API Reference for encode/encode_batch
2020-11-02 17:07:27 -05:00
f2f3ec51bd
Start the quicktour
2020-11-02 17:07:27 -05:00
3ee54766e3
Doc - Backbone for API Reference
2020-11-02 17:07:27 -05:00
7eb0a9255e
Doc - Basic layout - WIP
2020-11-02 17:07:27 -05:00
51cd5166d9
Doc - Uniform formatting + updated content
2020-11-02 17:07:27 -05:00
1da10c17ef
Doc - Fix default version
2020-11-02 17:07:27 -05:00
799954973d
Doc - Always go to latest when switching language
2020-11-02 17:07:27 -05:00
248586c8b9
Doc - Update deployed versions in custom.js
2020-11-02 17:07:27 -05:00
59d66c6db8
Doc - Add CI for automatic deployment
2020-11-02 17:07:27 -05:00
8e5d90d94d
Doc - Quick js/css update + remove sphinx_tabs deps
2020-11-02 17:07:27 -05:00
