mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-16 17:18:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
804 Commits 1 Branch 0 Tags
1b9ead7ca231da95b95f93381f00452f3fd1a15f
Commit Graph

309 Commits

Author SHA1 Message Date
Anthony MOI
f8d54edcdd Python - Fix cases where str expected instead of AddedToken 2020-03-25 19:22:53 -04:00
Anthony MOI
c65d53892d Python - Add bindings for new AddedToken options 2020-03-24 20:58:45 -04:00
Anthony MOI
d953d58cee Rust - Fix offsets when there are added tokens 2020-03-19 12:53:03 -04:00
Anthony MOI
d53de0e2da Python - Expose normalize on BaseTokenizer 2020-03-18 16:44:31 -04:00
Anthony MOI
ae0d330907 Update CHANGELOGs 2020-03-18 16:42:27 -04:00
Anthony MOI
60a4fb35f4 Python - Update bindings 2020-03-16 10:36:42 -04:00
Morgan Funtowicz
505bfbba82 Fix invalid error messages. 2020-03-12 15:38:29 +01:00
Morgan Funtowicz
5ed1f26c71 Throw a more meaningful error when provided python input is None. 2020-03-12 10:59:05 +01:00
Anthony MOI
257360acec Python - encode & encode batch with add_special_tokens 2020-03-10 16:21:10 -04:00
Anthony MOI
a9be177185 Update CHANGELOGs 2020-03-10 13:12:34 -04:00
Anthony MOI
28f022058c Keep default values as true 2020-03-10 12:58:53 -04:00
Anthony MOI
45f3eaaf72 Update bindings and typings 2020-03-10 12:28:24 -04:00
Anthony MOI
efbbfea558 Update ByteLevel PostProcessor 2020-03-10 12:05:04 -04:00
Anthony MOI
7e9003ccb7 Python - Update bindings 2020-03-09 18:37:03 -04:00
Anthony MOI
86d2e90ad2 Update CHANGELOGs 2020-03-06 17:44:44 -05:00
Anthony MOI
d778ed5e0a Python - Update README and implementation 2020-03-06 17:44:44 -05:00
Anthony MOI
52180a9179 Python - Add ByteLevel PostProcessor 2020-03-06 17:44:44 -05:00
Anthony MOI
b60eef5245 Python - Make style 2020-03-06 17:44:44 -05:00
Anthony MOI
d8e7a830b2 Update CHANGELOGs 2020-03-06 17:44:34 -05:00
Anthony MOI
b2e5f54b6f Python - Fix ByteLevelBPETokenizer implementation 2020-03-06 17:44:03 -05:00
Anthony MOI
f1460fadb9 Python - Update docs and implementations 2020-03-06 17:44:03 -05:00
Anthony MOI
2393506dc7 Python - Add ByteLevel Normalizer 2020-03-06 17:44:03 -05:00
Anthony MOI
47cef0e13a Python - Fix BPE and WordPiece builders usage 2020-03-06 12:20:39 -05:00
Anthony MOI
4b596e19dd Rust - Improve training progress for multiple files 2020-03-03 11:04:24 -05:00
Anthony MOI
8e791791d1 Python - prepare for release 2020-03-02 14:56:42 -05:00
Anthony MOI
4deeb9511f Update CHANGELOGs 2020-03-02 14:37:17 -05:00
Anthony MOI
f8f0702d98 Fix LongestFirst truncation strategy 2020-02-29 16:26:13 -05:00
Anthony MOI
657f8b6c15 Rust & Python - Update CHANGELOGs 2020-02-26 11:30:44 -05:00
Anthony MOI
3b10d640d5 Rust & Python - Update CHANGELOGs 2020-02-26 10:51:40 -05:00
Anthony MOI
2425fe877d Python - Update CHANGELOG 2020-02-26 09:31:17 -05:00
Anthony MOI
61b4c9c30a Python - Add missing tokens to BertWordPieceTokenizer 2020-02-26 09:21:54 -05:00
Anthony MOI
440e8e9bd9 Python - Bump version for release 2020-02-24 16:08:49 -05:00
Anthony MOI
be08d9574c Python - Add Changelog 2020-02-24 10:12:50 -05:00
Anthony MOI
999088ef94 Python - Bump version for release 2020-02-24 09:56:08 -05:00
Morgan Funtowicz
817b760ab9 Make name parameter Optional[str] on BaseTokenizer 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-02-22 14:57:43 +01:00
Morgan Funtowicz
d274a7691d Avoid breaking changes and let parameter name be Optional. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-02-22 14:56:59 +01:00
Morgan Funtowicz
0fc8be9d69 Formatting for python binding. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-02-22 00:17:44 +01:00
Morgan Funtowicz
f88a6b40ac Make parameter name on Model.save() optional. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-02-22 00:01:32 +01:00
Anthony MOI
11dd6c8bae Python - Bump version for release 2020-02-18 18:49:11 -05:00
Anthony MOI
41929462c7 Python - Add classifiers 2020-02-18 18:48:21 -05:00
Anthony MOI
d8a73c89a7 Python - Add Encoding length 2020-02-18 18:24:13 -05:00
Anthony MOI
d48fdbe057 Python - Only add special tokens when in-vocabulary 2020-02-18 17:27:27 -05:00
Anthony MOI
5daf1eea86 Python - Replace last BPETokenizer occurences 2020-02-18 16:25:59 -05:00
Anthony MOI
f263d7651f Python - RustFmt 2020-02-18 15:07:34 -05:00
Anthony MOI
8e9fae6be4 Python - Add check-style to Makefile 2020-02-18 11:11:07 -05:00
Anthony MOI
81be207819 Python - Black auto formatting 2020-02-18 10:45:36 -05:00
Anthony MOI
4706151c32 Python - Add Makefile with Black formatting 2020-02-18 10:45:10 -05:00
Anthony MOI
1509f747af Python - Uniformize implementations parameters 2020-02-18 10:27:10 -05:00
MOI Anthony
3512bd3400 Merge pull request #149 from colinclement/master 
Allow dropout option in ByteLevelBPETokenizer
 2020-02-18 09:59:40 -05:00
Morgan Funtowicz
891dd4adb8 Fix invalid num_added_tokens method call in BaseTokenizer. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-02-17 15:32:34 +01:00