mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-09 06:08:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
8585b761d16c5fb7ff26101fb2682c2f30e58e8f
tokenizers/bindings/python/tokenizers
History
Bjarte Johansen 6a4976ddd6 Implement __new__ for PreTokenizers 
__new__ allows PreTokenizers to be instansiated through the python
constructor.
2020-02-10 10:43:53 +01:00
..
decoders
Implement __new__ on Decoders
2020-02-10 10:43:53 +01:00
implementations
Implement __new__ for PreTokenizers
2020-02-10 10:43:53 +01:00
models
Introduce WordLevel model for TransformerXL (#125)
2020-02-05 16:51:35 +00:00
normalizers
Implement __new__ on Normalizers
2020-02-10 10:43:19 +01:00
pre_tokenizers
Implement __new__ for PreTokenizers
2020-02-10 10:43:53 +01:00
processors
Implement __new__ for PostProcessors
2020-02-10 10:43:53 +01:00
trainers
Implement __new__ on Trainers
2020-02-10 10:43:29 +01:00
__init__.py
Bump versions for release
2020-02-05 13:55:51 -05:00
__init__.pyi
fix :
2020-01-10 13:30:35 -05:00