Bjarte Johansen 6a4976ddd6 Implement __new__ for PreTokenizers ...

__new__ allows PreTokenizers to be instansiated through the python
constructor.

2020-02-10 10:43:53 +01:00

..

__init__.py

Improve flexibility in some Python binding (#107)

2020-02-03 10:41:33 +00:00

base_tokenizer.py

Merge pull request #99 from kdexd/get-vocab-size

2020-02-03 11:52:29 -05:00

bert_wordpiece.py

Implement __new__ for PreTokenizers

2020-02-10 10:43:53 +01:00

byte_level_bpe.py

Implement __new__ for PreTokenizers

2020-02-10 10:43:53 +01:00

char_level_bpe.py

Implement __new__ for PreTokenizers

2020-02-10 10:43:53 +01:00

sentencepiece_bpe.py

Implement __new__ for PreTokenizers

2020-02-10 10:43:53 +01:00