mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

master -> main (#1292)

Browse Source
This commit is contained in:
Arthit Suriyawongkul
2023-07-12 10:51:22 +01:00
committed by GitHub
parent 92bfb9c993
commit 150559b61e
1 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -5,7 +5,7 @@
<p> <p>
<p align="center"> <p align="center">
<img alt="Build" src="https://github.com/huggingface/tokenizers/workflows/Rust/badge.svg"> <img alt="Build" src="https://github.com/huggingface/tokenizers/workflows/Rust/badge.svg">
<a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE"> <a href="https://github.com/huggingface/tokenizers/blob/main/LICENSE">
<img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue&cachedrop"> <img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue&cachedrop">
</a> </a>
<a href="https://pepy.tech/project/tokenizers"> <a href="https://pepy.tech/project/tokenizers">
@ -30,9 +30,9 @@ versatility.
## Bindings ## Bindings
We provide bindings to the following languages (more to come!): We provide bindings to the following languages (more to come!):
- [Rust](https://github.com/huggingface/tokenizers/tree/master/tokenizers) (Original implementation) - [Rust](https://github.com/huggingface/tokenizers/tree/main/tokenizers) (Original implementation)
- [Python](https://github.com/huggingface/tokenizers/tree/master/bindings/python) - [Python](https://github.com/huggingface/tokenizers/tree/main/bindings/python)
- [Node.js](https://github.com/huggingface/tokenizers/tree/master/bindings/node) - [Node.js](https://github.com/huggingface/tokenizers/tree/main/bindings/node)
- [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo) - [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo)
## Quick example using Python: ## Quick example using Python: