Improve READMEs for from_pretrained

This commit is contained in:
Anthony Moi
2021-08-19 17:03:43 +02:00
committed by Anthony MOI
parent a4d0f3dd18
commit ad7090a5c7
4 changed files with 43 additions and 0 deletions

View File

@ -68,6 +68,14 @@ pip install setuptools_rust
python setup.py install
```
### Load a pretrained tokenizer from the Hub
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-cased")
```
### Using the provided Tokenizers
We provide some pre-build tokenizers to cover the most common cases. You can easily load one of