mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 16:25:30 +00:00
Update README.md (#1608)
This commit is contained in:
@ -27,6 +27,11 @@ versatility.
|
|||||||
original sentence that corresponds to a given token.
|
original sentence that corresponds to a given token.
|
||||||
- Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
|
- Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
|
||||||
|
|
||||||
|
## Performances
|
||||||
|
Performances can vary depending on hardware, but running the [~/bindings/python/benches/test_tiktoken.py](https://github.com/huggingface/tokenizers/bindings/python/benches/test_tiktoken.py) should give the following on a g6 aws instance:
|
||||||
|

|
||||||
|
|
||||||
|
|
||||||
## Bindings
|
## Bindings
|
||||||
|
|
||||||
We provide bindings to the following languages (more to come!):
|
We provide bindings to the following languages (more to come!):
|
||||||
|
Reference in New Issue
Block a user