mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Improve readme

Browse Source
This commit is contained in:
MOI Anthony
2020-01-10 11:29:11 -05:00
committed by GitHub
parent e7395285f2
commit 6295af6e6d
1 changed files with 20 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
README.md
View File

@ -1,4 +1,17 @@
# Tokenizers
<p align="center">
<br>
<img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
<br>
<p>
<p align="center">
<a href="https://circleci.com/gh/huggingface/transformers">
<img alt="Build" src="https://github.com/huggingface/tokenizers/workflows/Rust/badge.svg">
</a>
<a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE">
<img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue">
</a>
</p>
Provides an implementation of today's most used tokenizers, with a focus on performance and
versatility.
@ -14,6 +27,12 @@ versatility.
original sentence that corresponds to a given token.
- Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.
<p align="center">
<br>
<img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-repo-example.png" />
<br>
<p>
## Bindings
We provide bindings to the following languages (more to come!):