mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-06 20:58:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
789 Commits 1 Branch 0 Tags
def8333d45cd189eeb46ff8ff551c3d28510fc46
Commit Graph

13 Commits

Author SHA1 Message Date
Bjarte Johansen
823066fea9 Python - Update tests to use new models API 
- Check that new models have right subclass
 2020-04-06 21:40:08 +02:00
Anthony MOI
837791ee1f Python - Test BertWordPieceTokenizer 2020-04-01 17:25:56 -04:00
Anthony MOI
7fd7dfd113 Python - Test CharBPETokenizer 2020-04-01 17:25:56 -04:00
Anthony MOI
dbc23e20a9 Python - Test Models 2020-04-01 17:25:55 -04:00
Anthony MOI
53a7dbdaee Python - Test PostProcessors 2020-04-01 17:25:55 -04:00
Anthony MOI
a9f4c5950a Python - Test Decoders 2020-04-01 17:25:55 -04:00
Anthony MOI
0de9885da8 Python - Test PreTokenizers 2020-04-01 17:25:55 -04:00
Anthony MOI
d6692d4072 Python - Test Normalizers 2020-04-01 17:25:55 -04:00
Anthony MOI
3264ffe235 Python - Improve tests on Tokenizer 2020-04-01 17:25:55 -04:00
Anthony MOI
5ebe687753 Python - Add first implementations tests 2020-04-01 17:25:55 -04:00
Anthony MOI
023566fbbb Python - Add some tests utils 2020-04-01 17:25:55 -04:00
Anthony MOI
477037fd6b Python - Improve AddedToken repr 2020-04-01 17:25:55 -04:00
Anthony MOI
b055b77b54 Python - Add first tests: Tokenizer 2020-04-01 17:25:55 -04:00