mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 08:15:49 +00:00
Update README.md with install from source
This commit is contained in:
13
README.md
13
README.md
@ -40,6 +40,19 @@ We provide bindings to the following languages (more to come!):
|
||||
- [Node.js](https://github.com/huggingface/tokenizers/tree/main/bindings/node)
|
||||
- [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo)
|
||||
|
||||
## Installation
|
||||
|
||||
You can install from source using:
|
||||
```bash
|
||||
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
|
||||
```
|
||||
|
||||
our install the released versions with
|
||||
|
||||
```bash
|
||||
pip install tokenizers
|
||||
```
|
||||
|
||||
## Quick example using Python:
|
||||
|
||||
Choose your model between Byte-Pair Encoding, WordPiece or Unigram and instantiate a tokenizer:
|
||||
|
Reference in New Issue
Block a user