mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. (#1136)

Browse Source 
* [fix] Use unk_token

In SentencePieceBPETokenizer, when Vocab or  merges is None, unk_token cannot be used.

* [fix] If unk_token is None, this case is also considered.

* Update bindings/python/py_src/tokenizers/implementations/sentencepiece_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* [FIX] In CharBPETokenizer, Use unk_token.

In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used.

* Update bindings/python/py_src/tokenizers/implementations/char_level_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* Update bindings/python/py_src/tokenizers/implementations/char_level_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
This commit is contained in:
SeongBeomLEE
2022-12-27 19:13:52 +09:00
committed by GitHub
parent 60a00dda44
commit 9b155b5723
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/py_src/tokenizers/implementations/char_level_bpe.py
View File

@ -45,7 +45,7 @@ class CharBPETokenizer(BaseTokenizer):
)
)
else:
tokenizer = Tokenizer(BPE())
tokenizer = Tokenizer(BPE(unk_token=str(unk_token), dropout=dropout, end_of_word_suffix=suffix))
if tokenizer.token_to_id(str(unk_token)) is not None:
tokenizer.add_special_tokens([str(unk_token)])