mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
20 Commits 1 Branch 0 Tags
4e5106989fe79ca253202ef64cdc9a5df7a5e770
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Anthony MOI 4e5106989f Ability to load a BPE model from files
2019-11-18 10:00:53 -05:00
bindings
Node readme
2019-11-01 19:44:44 -04:00
tokenizers
Ability to load a BPE model from files
2019-11-18 10:00:53 -05:00
.gitignore
Global gitignore
2019-11-01 14:55:08 -04:00
README.md
Rust install
2019-11-01 19:45:00 -04:00

README.md

Tokenizers

# Install rust (if relevant)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
Description
No description provided
Readme Apache-2.0 7.4 MiB
Languages
Rust 72.3%
Python 20%
Jupyter Notebook 4.5%
TypeScript 2.3%
JavaScript 0.4%
Other 0.5%