mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 16:25:30 +00:00
committed by
Anthony MOI
parent
5cccaefcee
commit
558f2d8779
@ -70,8 +70,8 @@ print(output.tokens)
|
|||||||
# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]
|
# ["Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"]
|
||||||
```
|
```
|
||||||
|
|
||||||
Check the [python documentation](https://huggingface.co/tokenizers/python/latest/) or the
|
Check the [python documentation](https://huggingface.co/docs/tokenizers/python/latest) or the
|
||||||
[python quicktour](https://huggingface.co/tokenizers/python/latest/quicktour.html) to learn more!
|
[python quicktour](https://huggingface.co/docs/tokenizers/python/latest/quicktour.html) to learn more!
|
||||||
|
|
||||||
## Contributors
|
## Contributors
|
||||||
|
|
||||||
|
Reference in New Issue
Block a user