diff --git a/tokenizers/README.md b/tokenizers/README.md index cac7b586..be7f5eb0 100644 --- a/tokenizers/README.md +++ b/tokenizers/README.md @@ -137,3 +137,5 @@ fn main() -> Result<()> { **progressbar**: The progress bar visualization is enabled by default. It might be disabled if compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios) dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar. +**http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default. + With this feature enabled, `Tokenizer::from_pretrained` becomes accessible. diff --git a/tokenizers/src/lib.rs b/tokenizers/src/lib.rs index dd253e7f..44161271 100644 --- a/tokenizers/src/lib.rs +++ b/tokenizers/src/lib.rs @@ -125,6 +125,8 @@ //! **progressbar**: The progress bar visualization is enabled by default. It might be disabled if //! compilation for certain targets is not supported by the [termios](https://crates.io/crates/termios) //! dependency of the [indicatif](https://crates.io/crates/indicatif) progress bar. +//! **http**: This feature enables downloading the tokenizer via HTTP. It is disabled by default. +//! With this feature enabled, `Tokenizer::from_pretrained` becomes accessible. #[macro_use] extern crate log;