mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
1649d74536f3d9fa5e33d66eb68919e63d2272b4
tokenizers/bindings
History
SeongBeomLEE 9a25b2cb8e [FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. (#1120) 
* [fix] Use unk_token

In SentencePieceBPETokenizer, when Vocab or  merges is None, unk_token cannot be used.

* [fix] If unk_token is None, this case is also considered.

* Update bindings/python/py_src/tokenizers/implementations/sentencepiece_bpe.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2022-12-19 13:40:04 +01:00
..
node
Bump decode-uri-component from 0.2.0 to 0.2.2 in /bindings/node (#1116)
2022-12-05 18:09:38 +01:00
python
[FIX] In SentencePieceBPETokenizer, when Vocab or merges is None, unk_token cannot be used. (#1120)
2022-12-19 13:40:04 +01:00