mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 16:49:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Nicolas Patry
2020-11-02 16:07:35 +01:00
committed by Anthony MOI
parent d788a950ac
commit b35d514785
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -54,7 +54,7 @@ from tokenizers.pre_tokenizers import Whitespace
tokenizer.pre_tokenizer = Whitespace() tokenizer.pre_tokenizer = Whitespace()
``` ```
Then training your tokenizer on a set of files jsut takes two lines of codes: Then training your tokenizer on a set of files just takes two lines of codes:
```python ```python
from tokenizers.trainers import BpeTrainer from tokenizers.trainers import BpeTrainer