mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update documentation of Rust feature (#1711)

Browse Source 
* Update documentation of Rust feature

* Synchronize README.md and src/lib.rs
This commit is contained in:
sondalex
2025-01-09 12:08:45 +01:00
committed by GitHub
parent bdfc38b78d
commit c04b97aab1
2 changed files with 4 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tokenizers/README.md
View File

@ -137,3 +137,5 @@ fn main() -> Result<()> {
**progressbar**: The progress bar visualization is enabled by default. It might be disabled if **progressbar**: The progress bar visualization is enabled by default. It might be disabled if
compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios) compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios)
dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar. dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar.
**http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default.
With this feature enabled, `Tokenizer::from_pretrained` becomes accessible.

2
tokenizers/src/lib.rs
View File

@ -125,6 +125,8 @@
//! **progressbar**: The progress bar visualization is enabled by default. It might be disabled if //! **progressbar**: The progress bar visualization is enabled by default. It might be disabled if
//! compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios) //! compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios)
//! dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar. //! dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar.
//! **http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default.
//! With this feature enabled, `Tokenizer::from_pretrained` becomes accessible.
#[macro_use] #[macro_use]
extern crate log; extern crate log;