आलोक 6fe284dd8d Use supplied UNK token even when vocab absent ...

If a vocab file isn't provided the supplied unk token (different from [UNK]) gets ignored and later throws an error:
Exception: WordPiece error: Missing [UNK] token from the vocabulary
when trying to encode an input string with an unknown token.

2020-07-05 19:01:04 +05:30

..

decoders

Python - Bindings for Wordpiece decoder's cleanup

2020-02-13 17:50:37 -05:00

implementations

Use supplied UNK token even when vocab absent

2020-07-05 19:01:04 +05:30

models

Python - Update types with new models API

2020-04-06 21:40:08 +02:00

normalizers

Python - Update bindings

2020-03-09 18:37:03 -04:00

pre_tokenizers

Python - Update bindings

2020-03-09 18:37:03 -04:00

processors

Python - Add offsets trimming to RobertaProcessing

2020-04-15 18:49:38 -04:00

trainers

Python - Add bindings for new AddedToken options

2020-03-24 20:58:45 -04:00

__init__.py

Python - Bump version for 0.8.0 release

2020-06-26 16:12:26 -04:00

__init__.pyi

Python - Improve AddedToken interface

2020-06-19 17:53:46 -04:00