mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-10 22:58:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
555 Commits 1 Branch 0 Tags
f263d7651fb75e7b1b6c64d7779be42bc085091c
Commit Graph

555 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Pierric Cistac
66d65595f6 clean package / package-lock 2020-01-10 11:53:30 -05:00
Pierric Cistac
6b0935d5de first implementations draft 2020-01-10 11:53:30 -05:00
Pierric Cistac
63532ef583 move native bindings typings into subdir and reexport from root index 
+ switch to typescript to prepare for wrappers
 2020-01-10 11:53:30 -05:00
Pierric Cistac
d56f719cbb better structure 2020-01-10 11:53:30 -05:00
Pierric Cistac
0b8a51c010 First draft node typings 2020-01-10 11:53:30 -05:00
Anthony MOI
0925c30997 Node - Improve handling of optionals 2020-01-10 11:52:15 -05:00
MOI Anthony
15739d5e7e Readme - remove bad link 2020-01-10 11:38:15 -05:00
MOI Anthony
3ff78e43aa Add rust to readme 2020-01-10 11:32:25 -05:00
MOI Anthony
b4701773b5 Tweak readme 2020-01-10 11:29:43 -05:00
MOI Anthony
6295af6e6d Improve readme 2020-01-10 11:29:11 -05:00
Anthony MOI
e7395285f2 Split readme 2020-01-10 11:09:28 -05:00
Anthony MOI
b27737d97c Python - Typings update 2020-01-10 10:06:24 -05:00
MOI Anthony
b357a3ed5a Merge pull request #48 from huggingface/fix-python-stuff 
Fix a few python stuff
 2020-01-10 10:03:02 -05:00
Anthony MOI
07e2548e01 Quick tweak of the training progress bar 2020-01-10 10:00:54 -05:00
thomwolf
d8f3fba245 fix training and wordpiece 2020-01-10 10:47:50 +01:00
thomwolf
1a802cb484 fix typos 2020-01-10 10:47:36 +01:00
Anthony MOI
d46ea842c2 Python - IndexableString accepts tuples directly 2020-01-10 00:32:30 -05:00
Anthony MOI
1f16fcbe77 Show progress while reading files during training 2020-01-10 00:21:46 -05:00
Anthony MOI
7e59ff8ee9 Node - Add missing getters and setters on Tokenizer 2020-01-09 21:51:16 -05:00
Anthony MOI
fdb67e02ff Node - Tokenizer can be trained 2020-01-09 21:01:57 -05:00
Anthony MOI
a2c16c71e9 Node - Add trainers 2020-01-09 20:12:14 -05:00
Anthony MOI
ddbc0491bd Node - Add missing models 2020-01-09 19:24:40 -05:00
Anthony MOI
b75577eecc Node - Add pre tokenizers 2020-01-09 19:14:15 -05:00
Anthony MOI
264cdb4266 Node - Add normalizers 2020-01-09 18:37:10 -05:00
Anthony MOI
796601adbc Node - Add addTokens and addSpecialTokens 2020-01-09 17:41:42 -05:00
Anthony MOI
a1fd99125c Node - tokenToId & idToToken 2020-01-09 17:23:02 -05:00
Anthony MOI
63a3ffbf13 Node - Add decode & decodeBatch 2020-01-09 17:15:01 -05:00
Anthony MOI
6816628d1a Node - Hotfix EncodeTask 2020-01-09 16:05:48 -05:00
Anthony MOI
cb52b71f63 Node - Fix tasks count 2020-01-09 15:43:16 -05:00
Anthony MOI
6561511214 Node - Pad & Truncate on Encoding 2020-01-09 15:43:07 -05:00
Anthony MOI
778b611fb5 Node - Add some Encoding features 2020-01-09 14:50:39 -05:00
Anthony MOI
274fcd3bfe Node - Lift running tasks restriction 
Immutable actions like encoding shouldn't be prevented. Mutable one though must be.
 2020-01-09 14:18:43 -05:00
Anthony MOI
de35bb4f45 Node - Fix EncodeTask 2020-01-09 14:04:20 -05:00
Anthony MOI
bda03ffe8c Node - Make encode & encodeBatch async 2020-01-09 13:49:59 -05:00
Anthony MOI
19d41a5810 Node - Add encode & encodeBatch 2020-01-09 11:47:35 -05:00
Anthony MOI
1a54692190 Node - Add PostProcessors 2020-01-09 01:41:41 -05:00
Anthony MOI
83f21ab33d Node - Fix typo 2020-01-09 01:08:54 -05:00
Anthony MOI
afb6b48361 Node - Expose decoders 2020-01-09 01:05:36 -05:00
Anthony MOI
3685eb1809 Node - Improve namings 2020-01-09 01:02:28 -05:00
Anthony MOI
156d86d91e Node - Basic Tokenizer + BPE 2020-01-09 00:04:53 -05:00
MOI Anthony
13f3fbed30 Merge pull request #47 from huggingface/sentencepiece_export 
Added SentencePiece and YouTokenToMe model extractors.
 2020-01-08 17:18:20 -05:00
Morgan Funtowicz
be10f542ce Added SentencePiece and YouTokenToMe model extractors. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-01-08 22:55:00 +01:00
Anthony MOI
f86b8d412b Fix NormalizedString split_off bis2 2020-01-08 16:51:59 -05:00
Anthony MOI
7ac45472b6 Fix NormalizedString split_off bis 2020-01-08 16:48:47 -05:00
Anthony MOI
3b2e19f52c Fix NormalizedString split_off 2020-01-08 16:43:07 -05:00
MOI Anthony
313d674dc0 Merge pull request #45 from huggingface/bpe_save_compat_tweak 
BPE save compatibility tweak
 2020-01-08 16:23:15 -05:00
Anthony MOI
3af2a43cae Hotfix Python bindings 2020-01-08 16:20:05 -05:00
Anthony MOI
ef21c9a7b0 Hotfix for new Builder 
cc @epwalsh
 2020-01-08 16:19:51 -05:00
Julien Chaumond
6697d65544 Fixup + better compat 2020-01-08 20:52:28 +00:00
Evan Pete Walsh
d2d5b1eae7 tweak builder pattern to find defaults in Builder::Default (#42) 
* alternate builder pattern

* BpeBuilder

* update BpeTrainerBuilder
 2020-01-08 12:10:34 -08:00