mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-24 09:09:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,719 Commits 1 Branch 0 Tags
058e34b4216c22a867f739e131e8280f9968ff12
Commit Graph

3 Commits

Author SHA1 Message Date
Nicolas Patry
6113666624 Updating python formatting. (#1079) 
* Updating python formatting.

* Forgot gh action.

* Skipping isort to prevent circular imports.

* Updating stub.

* Removing `isort` (it contradicts `stub.py`).

* Fixing weird stub black/isort disagreeement.
 2022-10-05 15:29:33 +02:00
SaulLu
da4c7b10e4 Add a way to specify the unknown token in SentencePieceUnigramTokenizer python implem (#762) 
* add a way to specify the unknown token in `SentencePieceUnigramTokenizer`

* add test that verify that an exception is raised for the missing unknown token

* style

* add test tokens
 2021-08-12 09:42:44 -04:00
Anthony MOI
d94fa220b6 Python - Add train_from_iterator to implementations 2021-01-07 09:02:20 -05:00