mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-31 12:39:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #75 from huggingface/doc_parallelism

Browse Source 
Added doc for setting tokenizers level of parallelism.
This commit is contained in:
MOI Anthony
2020-01-15 09:42:22 -05:00
committed by GitHub
parent 65b35385f8 11fb79cee8
commit 7d10dd0fd6
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tokenizers/README.md
View File

@ -54,3 +54,10 @@ fn main() -> Result<()> {
Ok(()) Ok(())
} }
``` ```
## Additional information
- tokenizers is designed to leverage CPU parallelism when possible. The level of parallelism is determined
by the total number of core/threads your CPU provides but this can be tuned by setting the `RAYON_RS_NUM_CPUS`
environment variable. As an example setting `RAYON_RS_NUM_CPUS=4` will allocate a maximum of 4 threads.
**_Please note this behavior may evolve in the future_**