mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 05:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Some suggestions from @narsil

Browse Source
This commit is contained in:
Anthony MOI
2020-09-22 11:59:40 -04:00
committed by Anthony MOI
parent 31b81f109b
commit b24a2fc178
6 changed files with 60 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
bindings/python/py_src/tokenizers/__init__.pyi
View File

@@ -73,11 +73,11 @@ class PreTokenizedString:
The string sequence used to initialize this PreTokenizedString
"""
pass
def split(self, func: Callable[[NormalizedString], List[NormalizedString]]):
def split(self, func: Callable[[index, NormalizedString], List[NormalizedString]]):
""" Split the PreTokenizedString using the given `func`
Args:
func: Callable[[NormalizedString], List[NormalizedString]]:
func: Callable[[index, NormalizedString], List[NormalizedString]]:
The function used to split each underlying split.
It is expected to return a list of `NormalizedString`, that represent the new
splits. If the given `NormalizedString` does not need any splitting, we can