mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-24 00:59:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python readme

Browse Source
This commit is contained in:
Anthony MOI
2019-11-01 19:42:36 -04:00
parent 6d91bf4005
commit 05cbb32eca
5 changed files with 18 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
bindings/python/example.py Normal file
View File

@ -0,0 +1,3 @@
from tokenizers import WhitespaceTokenizer
print(WhitespaceTokenizer.tokenize("Hey man!"))