mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 08:15:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md with install from source

Browse Source
This commit is contained in:
Arthur
2024-11-15 21:51:39 +01:00
committed by GitHub
parent cc5fb01a2f
commit f0c48bd89a
1 changed files with 13 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
README.md
View File

@ -39,6 +39,19 @@ We provide bindings to the following languages (more to come!):
- [Python](https://github.com/huggingface/tokenizers/tree/main/bindings/python) - [Python](https://github.com/huggingface/tokenizers/tree/main/bindings/python)
- [Node.js](https://github.com/huggingface/tokenizers/tree/main/bindings/node) - [Node.js](https://github.com/huggingface/tokenizers/tree/main/bindings/node)
- [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo) - [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo)
## Installation
You can install from source using:
```bash
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
```
our install the released versions with
```bash
pip install tokenizers
```
## Quick example using Python: ## Quick example using Python: