mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README

Browse Source
This commit is contained in:
Anthony MOI
2019-12-03 17:26:20 -05:00
parent 75232c0f06
commit c46ec97855
1 changed files with 7 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
bindings/python/README.md
View File

@ -12,16 +12,18 @@ really often in the coming days/weeks, so use at your own risks.
### Installation ### Installation
#### With pip: #### With pip:
```
```bash
pip install tokenizers pip install tokenizers
``` ```
#### From sources: #### From sources:
To use this method, you need to have the Rust nightly toolchain installed. To use this method, you need to have the Rust nightly toolchain installed.
```
```bash
# Install with: # Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -default-toolchain nightly-2019-11-01 - curl https://sh.rustup.rs -sSf | sh -s -- -default-toolchain nightly-2019-11-01 -y
export PATH="$HOME/.cargo/bin:$PATH" export PATH="$HOME/.cargo/bin:$PATH"
# Or select the right toolchain: # Or select the right toolchain:
@ -29,7 +31,8 @@ rustup default nightly-2019-11-01
``` ```
Once Rust is installed and using the right toolchain you can do the following. Once Rust is installed and using the right toolchain you can do the following.
```
```bash
git clone https://github.com/huggingface/tokenizers git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python cd tokenizers/bindings/python