mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-07 05:08:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
804 Commits 1 Branch 0 Tags
1b9ead7ca231da95b95f93381f00452f3fd1a15f
Commit Graph

10 Commits

Author SHA1 Message Date
Bjarte Johansen
2dc48e56ac Python - Update pyo3 version 
* Use __new__ instead of static method as model constructors
 2020-04-06 21:20:16 +02:00
Anthony MOI
c65d53892d Python - Add bindings for new AddedToken options 2020-03-24 20:58:45 -04:00
Anthony MOI
08ce105195 Python - Hotfix WordPieceTrainer constructor 2020-02-11 08:13:57 -05:00
Bjarte Johansen
4971e9608d Implement __new__ on Trainers 
__new__ allows Trainers to be initialized in the normal python
fashion.
 2020-02-10 10:43:29 +01:00
Anthony MOI
ef21c9a7b0 Hotfix for new Builder 
cc @epwalsh
 2020-01-08 16:19:51 -05:00
Anthony MOI
c51e340492 Python - Add WordPieceTrainer 2020-01-03 19:37:29 -05:00
Anthony MOI
e64b54b29e Python - Update BpeTrainer interface 2020-01-03 19:37:29 -05:00
Anthony MOI
0589deb6e2 Python - Expose BpeTrainer options 2020-01-02 18:09:04 -05:00
epwalsh
c0ed873c4d simplify initialization of BpeTrainer 2019-12-23 20:13:48 -05:00
Anthony MOI
eaafb22511 Add bindings for Trainer in Python 2019-12-03 15:54:15 -05:00