mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-16 17:18:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Improve Rust readme header

Browse Source
This commit is contained in:
MOI Anthony
2020-01-10 12:15:29 -05:00
committed by GitHub
parent 1fa11eba27
commit d26db40da7
1 changed files with 13 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
tokenizers/README.md
View File

@@ -1,4 +1,16 @@
# Tokenizers
<p align="center">
<br>
<img src="https://huggingface.co/landing/assets/tokenizers/tokenizers-logo.png" width="600"/>
<br>
<p>
<p align="center">
<img alt="Build" src="https://github.com/huggingface/tokenizers/workflows/Rust/badge.svg">
<a href="https://github.com/huggingface/tokenizers/blob/master/LICENSE">
<img alt="GitHub" src="https://img.shields.io/github/license/huggingface/tokenizers.svg?color=blue">
</a>
</p>
<br>
The core of `tokenizers`, written in Rust.
Provides an implementation of today's most used tokenizers, with a focus on performance and