SeongBeomLEE 9b155b5723 [FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. (#1136) ...

* [fix] Use unk_token

In SentencePieceBPETokenizer, when Vocab or  merges is None, unk_token cannot be used.

* [fix] If unk_token is None, this case is also considered.

* Update bindings/python/py_src/tokenizers/implementations/sentencepiece_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* [FIX] In CharBPETokenizer, Use unk_token.

In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used.

* Update bindings/python/py_src/tokenizers/implementations/char_level_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* Update bindings/python/py_src/tokenizers/implementations/char_level_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

2022-12-27 11:13:52 +01:00

..

node

Ignore Cargo.lock for subfolders (#1131)

2022-12-25 11:35:47 +01:00

python

[FIX] In CharBPETokenizer, when Vocab or merges is None, unk_token cannot be used. (#1136)

2022-12-27 11:13:52 +01:00