mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-16 17:18:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
519 Commits 1 Branch 0 Tags
f32e0c09fcb692c0f6f9cac2fa94cb1557292d1b
Commit Graph

519 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Anthony MOI
dd569020c1 Bump python version for release 2020-01-10 13:49:26 -05:00
MOI Anthony
613a25f7b3 Merge pull request #49 from huggingface/fix-pyi 
Fix Python typings
 2020-01-10 13:41:09 -05:00
Anthony MOI
89e0d90c8a Python - Final fix of the typings 2020-01-10 13:30:35 -05:00
Pierric Cistac
56878a8e43 fix : 2020-01-10 13:30:35 -05:00
Pierric Cistac
958883af74 fix imports in root __init__.pyi 2020-01-10 13:30:35 -05:00
MOI Anthony
4cfbb11d18 Update README 2020-01-10 12:55:10 -05:00
MOI Anthony
ac2945143d Add doc badge 2020-01-10 12:42:49 -05:00
Anthony MOI
aa0b644c11 Cargo.toml fix number of keywords 2020-01-10 12:30:42 -05:00
Anthony MOI
46e0ad5c9a Bump for crates.io 2020-01-10 12:29:00 -05:00
MOI Anthony
b491c0b8c4 Update Python Readme 2020-01-10 12:18:16 -05:00
MOI Anthony
d26db40da7 Improve Rust readme header 2020-01-10 12:15:29 -05:00
Anthony MOI
1fa11eba27 Quick example of rust usage 2020-01-10 12:13:49 -05:00
Pierric Cistac
34875d5771 new batch of typings 2020-01-10 11:53:30 -05:00
Pierric Cistac
969d994f70 add more methods on tokenizer 2020-01-10 11:53:30 -05:00
Pierric Cistac
66d65595f6 clean package / package-lock 2020-01-10 11:53:30 -05:00
Pierric Cistac
6b0935d5de first implementations draft 2020-01-10 11:53:30 -05:00
Pierric Cistac
63532ef583 move native bindings typings into subdir and reexport from root index 
+ switch to typescript to prepare for wrappers
 2020-01-10 11:53:30 -05:00
Pierric Cistac
d56f719cbb better structure 2020-01-10 11:53:30 -05:00
Pierric Cistac
0b8a51c010 First draft node typings 2020-01-10 11:53:30 -05:00
Anthony MOI
0925c30997 Node - Improve handling of optionals 2020-01-10 11:52:15 -05:00
MOI Anthony
15739d5e7e Readme - remove bad link 2020-01-10 11:38:15 -05:00
MOI Anthony
3ff78e43aa Add rust to readme 2020-01-10 11:32:25 -05:00
MOI Anthony
b4701773b5 Tweak readme 2020-01-10 11:29:43 -05:00
MOI Anthony
6295af6e6d Improve readme 2020-01-10 11:29:11 -05:00
Anthony MOI
e7395285f2 Split readme 2020-01-10 11:09:28 -05:00
Anthony MOI
b27737d97c Python - Typings update 2020-01-10 10:06:24 -05:00
MOI Anthony
b357a3ed5a Merge pull request #48 from huggingface/fix-python-stuff 
Fix a few python stuff
 2020-01-10 10:03:02 -05:00
Anthony MOI
07e2548e01 Quick tweak of the training progress bar 2020-01-10 10:00:54 -05:00
thomwolf
d8f3fba245 fix training and wordpiece 2020-01-10 10:47:50 +01:00
thomwolf
1a802cb484 fix typos 2020-01-10 10:47:36 +01:00
Anthony MOI
d46ea842c2 Python - IndexableString accepts tuples directly 2020-01-10 00:32:30 -05:00
Anthony MOI
1f16fcbe77 Show progress while reading files during training 2020-01-10 00:21:46 -05:00
Anthony MOI
7e59ff8ee9 Node - Add missing getters and setters on Tokenizer 2020-01-09 21:51:16 -05:00
Anthony MOI
fdb67e02ff Node - Tokenizer can be trained 2020-01-09 21:01:57 -05:00
Anthony MOI
a2c16c71e9 Node - Add trainers 2020-01-09 20:12:14 -05:00
Anthony MOI
ddbc0491bd Node - Add missing models 2020-01-09 19:24:40 -05:00
Anthony MOI
b75577eecc Node - Add pre tokenizers 2020-01-09 19:14:15 -05:00
Anthony MOI
264cdb4266 Node - Add normalizers 2020-01-09 18:37:10 -05:00
Anthony MOI
796601adbc Node - Add addTokens and addSpecialTokens 2020-01-09 17:41:42 -05:00
Anthony MOI
a1fd99125c Node - tokenToId & idToToken 2020-01-09 17:23:02 -05:00
Anthony MOI
63a3ffbf13 Node - Add decode & decodeBatch 2020-01-09 17:15:01 -05:00
Anthony MOI
6816628d1a Node - Hotfix EncodeTask 2020-01-09 16:05:48 -05:00
Anthony MOI
cb52b71f63 Node - Fix tasks count 2020-01-09 15:43:16 -05:00
Anthony MOI
6561511214 Node - Pad & Truncate on Encoding 2020-01-09 15:43:07 -05:00
Anthony MOI
778b611fb5 Node - Add some Encoding features 2020-01-09 14:50:39 -05:00
Anthony MOI
274fcd3bfe Node - Lift running tasks restriction 
Immutable actions like encoding shouldn't be prevented. Mutable one though must be.
 2020-01-09 14:18:43 -05:00
Anthony MOI
de35bb4f45 Node - Fix EncodeTask 2020-01-09 14:04:20 -05:00
Anthony MOI
bda03ffe8c Node - Make encode & encodeBatch async 2020-01-09 13:49:59 -05:00
Anthony MOI
19d41a5810 Node - Add encode & encodeBatch 2020-01-09 11:47:35 -05:00
Anthony MOI
1a54692190 Node - Add PostProcessors 2020-01-09 01:41:41 -05:00