mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README

Browse Source
This commit is contained in:
Anthony MOI
2019-12-03 17:26:20 -05:00
parent 75232c0f06
commit c46ec97855
1 changed files with 7 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
bindings/python/README.md
View File

@ -12,16 +12,18 @@ really often in the coming days/weeks, so use at your own risks.
### Installation
#### With pip:
```
```bash
pip install tokenizers
```
#### From sources:
To use this method, you need to have the Rust nightly toolchain installed.
```
```bash
# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -default-toolchain nightly-2019-11-01 -
curl https://sh.rustup.rs -sSf | sh -s -- -default-toolchain nightly-2019-11-01 -y
export PATH="$HOME/.cargo/bin:$PATH"
# Or select the right toolchain:
@ -29,7 +31,8 @@ rustup default nightly-2019-11-01
```
Once Rust is installed and using the right toolchain you can do the following.
```
```bash
git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python