mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 13:48:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add SplitDelimiterBehavior to Punctuation constructor (#657)

Browse Source 
Resolves: #642
This commit is contained in:
Vlad Artamonov
2021-08-13 09:19:23 -04:00
committed by GitHub
parent c1100dcbe3
commit e2bf8daa3a
10 changed files with 69 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi
View File

@@ -308,10 +308,16 @@ class Metaspace(PreTokenizer):
class Punctuation(PreTokenizer):
"""
This pre-tokenizer simply splits on punctuation as individual characters.`
This pre-tokenizer simply splits on punctuation as individual characters.
Args:
behavior (:class:`~tokenizers.SplitDelimiterBehavior`):
The behavior to use when splitting.
Choices: "removed", "isolated" (default), "merged_with_previous", "merged_with_next",
"contiguous"
"""
def __init__(self):
def __init__(self, behavior="isolated"):
pass
def pre_tokenize(self, pretok):
"""