mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 13:48:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python - Make style

Browse Source
This commit is contained in:
Anthony MOI
2020-03-05 11:15:09 -05:00
parent 43811698a1
commit b60eef5245
1 changed files with 0 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
bindings/python/tokenizers/pre_tokenizers/__init__.pyi
View File

@@ -23,7 +23,6 @@ class ByteLevel(PreTokenizer):
def __init__(self) -> None: def __init__(self) -> None:
""" Instantiate a new ByteLevel PreTokenizer """ """ Instantiate a new ByteLevel PreTokenizer """
pass pass
@staticmethod @staticmethod
def alphabet() -> List[str]: def alphabet() -> List[str]:
""" Returns the alphabet used by this PreTokenizer. """ Returns the alphabet used by this PreTokenizer.