mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-05 04:08:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

TemplateProcessing - Add @narsil suggestions

Browse Source 
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
This commit is contained in:
Anthony MOI
2020-09-10 14:51:39 -04:00
committed by Anthony MOI
parent b7df6539e6
commit fee1d4e8a3
2 changed files with 42 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
bindings/python/py_src/tokenizers/processors/__init__.pyi
View File

@@ -129,6 +129,11 @@ class TemplateProcessing(PostProcessor):
Note that we are saying the "default" type_id because each SpecialToken can define
its own type_id which would override the provided default.
**Warning**: You must ensure that you are giving the correct tokens/ids as these
will be added to the Encoding without any further check. If the given ids correspond
to something totally different in a `Tokenizer` using this `PostProcessor`, it
might lead to unexpected results.
"""
def __init__(self, seq_a: Template, seq_b: Template, special_tokens: Tokens) -> None: