mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Adding a link to the Ruby port of tokenizers (#961)

Browse Source
This commit is contained in:
Nicolas Patry
2022-03-24 17:09:30 +01:00
committed by GitHub
parent 28fe0e40e7
commit 20ec60aeba
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
README.md
View File

@ -33,6 +33,7 @@ We provide bindings to the following languages (more to come!):
- [Rust](https://github.com/huggingface/tokenizers/tree/master/tokenizers) (Original implementation)
- [Python](https://github.com/huggingface/tokenizers/tree/master/bindings/python)
- [Node.js](https://github.com/huggingface/tokenizers/tree/master/bindings/node)
- [Ruby](https://github.com/ankane/tokenizers-ruby) (Contributed by @ankane, external repo)
## Quick example using Python: