mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-04 11:48:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,262 Commits 1 Branch 0 Tags
a2289d49b4fc7554b0c5d8e1111229fd77e254fc
Commit Graph

3 Commits

Author SHA1 Message Date
Nicolas Patry
6887c0f04d Black pass. 2020-08-31 14:05:39 -04:00
Sławomir Dadas
0865a9ad55 Python - improve compatibility with sentencepiece in the conversion script 2020-04-11 17:35:50 +02:00
Morgan Funtowicz
be10f542ce Added SentencePiece and YouTokenToMe model extractors. 
Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>
 2020-01-08 22:55:00 +01:00