mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-07 13:18:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
789 Commits 1 Branch 0 Tags
def8333d45cd189eeb46ff8ff551c3d28510fc46
Commit Graph

789 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Bjarte Johansen
def8333d45 Python - Update changelog 2020-04-06 21:40:23 +02:00
Bjarte Johansen
fab97475e5 Python - Update examples to use new models API 2020-04-06 21:40:23 +02:00
Bjarte Johansen
823066fea9 Python - Update tests to use new models API 
- Check that new models have right subclass
 2020-04-06 21:40:08 +02:00
Bjarte Johansen
38bc788002 Python - Update implementations to use new API 2020-04-06 21:40:08 +02:00
Bjarte Johansen
69ed81e618 Python - Update types with new models API 2020-04-06 21:40:08 +02:00
Bjarte Johansen
2dc48e56ac Python - Update pyo3 version 
* Use __new__ instead of static method as model constructors
 2020-04-06 21:20:16 +02:00
Anthony MOI
2a4e5f81de Merge pull request #213 from huggingface/python-tests 
Add Python tests
 2020-04-02 14:09:22 -04:00
Anthony MOI
b03fea1d66 Python - Update workflow and Makefile with tests 2020-04-01 17:36:33 -04:00
Anthony MOI
837791ee1f Python - Test BertWordPieceTokenizer 2020-04-01 17:25:56 -04:00
Anthony MOI
7fd7dfd113 Python - Test CharBPETokenizer 2020-04-01 17:25:56 -04:00
Anthony MOI
dbc23e20a9 Python - Test Models 2020-04-01 17:25:55 -04:00
Anthony MOI
53a7dbdaee Python - Test PostProcessors 2020-04-01 17:25:55 -04:00
Anthony MOI
a9f4c5950a Python - Test Decoders 2020-04-01 17:25:55 -04:00
Anthony MOI
0de9885da8 Python - Test PreTokenizers 2020-04-01 17:25:55 -04:00
Anthony MOI
d6692d4072 Python - Test Normalizers 2020-04-01 17:25:55 -04:00
Anthony MOI
3264ffe235 Python - Improve tests on Tokenizer 2020-04-01 17:25:55 -04:00
Anthony MOI
5ebe687753 Python - Add first implementations tests 2020-04-01 17:25:55 -04:00
Anthony MOI
023566fbbb Python - Add some tests utils 2020-04-01 17:25:55 -04:00
Anthony MOI
477037fd6b Python - Improve AddedToken repr 2020-04-01 17:25:55 -04:00
Anthony MOI
b055b77b54 Python - Add first tests: Tokenizer 2020-04-01 17:25:55 -04:00
Pierric Cistac
d3fb1d12f4 Try avoid duplicated github actions in PRs 2020-04-01 16:39:51 -04:00
Pierric Cistac
fd0c349d09 Micro tweak changelog 2020-04-01 14:30:01 -04:00
Pierric Cistac
cca20f4912 Node - Version 0.6.1 2020-04-01 14:26:39 -04:00
Anthony MOI
449222c659 Update Python workflow to help find right nightly 
Whenever a component is missing from the last nightly (here rustfmt)
this should help find the last nightly that did have it.
 2020-04-01 14:16:36 -04:00
Pierric Cistac
d90593a5e8 Run github actions on pull requests 
Try to fix actions not running for pull requests opened by external contributors cc @n1t0
 2020-04-01 14:04:14 -04:00
Qiang Kou
85488dd633 Fix typos in JsAddedToken init in Node binding 2020-04-01 13:47:26 -04:00
Anthony MOI
f15c088cf3 Python - Hotfix typing import 2020-04-01 11:35:07 -04:00
Anthony MOI
2a84ef12cf Python - Add missing get_vocab from BaseTokenizer 2020-04-01 11:32:54 -04:00
Anthony MOI
b770f36428 Rust - Hotfix special tokens with wrong id 2020-04-01 11:02:55 -04:00
Anthony MOI
93a83127ae Bump version for Python release 2020-03-31 14:25:47 -04:00
Morgan Funtowicz
afe9cfe96e Strip should inherits from Normalizer on Python binding. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-03-31 20:20:09 +02:00
Pierric Cistac
6e89d97c0b Node - tweak readme 2020-03-30 14:25:18 -04:00
Pierric Cistac
bb920af9e8 Node - Version 0.6.0 2020-03-30 12:38:15 -04:00
Anthony MOI
2b74bef6b2 Rust - Ignore data folder 2020-03-30 12:36:29 -04:00
Anthony MOI
279db8537e Update CHANGELOGs 2020-03-27 18:46:20 -04:00
Anthony MOI
3f4a6b746b Rust - BPEDecoder now trims after decoding 
Fix #205
 2020-03-27 17:58:31 -04:00
Anthony MOI
5038a7f74e Update CHANGELOGs 2020-03-27 17:49:02 -04:00
Anthony MOI
2a71a63242 Merge pull request #208 from huggingface/get-vocab 
Add the ability to get the vocabulary
 2020-03-27 17:38:03 -04:00
Anthony MOI
ca2510a44f Node - expose getVocab on Tokenizer 2020-03-27 16:07:35 -04:00
Anthony MOI
a2a6d80017 Python - expost get_vocab on Tokenizer 2020-03-27 11:53:18 -04:00
Anthony MOI
e191008751 Rust - expose get_vocab on Tokenizer & Model 2020-03-27 10:58:51 -04:00
Anthony MOI
e8aec7a624 Bump version for Python release 2020-03-27 09:17:35 -04:00
Anthony MOI
8783bd1023 Merge pull request #206 from huggingface/hotfix-py-missing-import 
Added missing imports for AddedToken
 2020-03-27 09:16:01 -04:00
Morgan Funtowicz
bc46064a61 Added missing imports for AddedToken 2020-03-27 12:32:23 +01:00
Anthony MOI
b132be34af Bump version for Python release 2020-03-26 17:26:14 -04:00
Anthony MOI
ab12a7fbac Bump version for Rust release 2020-03-26 17:21:55 -04:00
Anthony MOI
73ad9a52af hotfix CHANGELOG - missing from previous commit 2020-03-26 17:18:36 -04:00
Anthony MOI
6c232886b8 Improve & update CHANGELOGs 2020-03-26 17:08:42 -04:00
Anthony MOI
1bdff71362 Merge pull request #200 from huggingface/encoding-mappings 
New mappings on Encoding
 2020-03-26 15:58:06 -04:00
Pierric Cistac
e9667a7b83 Node - tokenizer.postProcess bindings 2020-03-26 15:42:45 -04:00