diff --git a/docs/source-doc-builder/api/added-tokens.mdx b/docs/source-doc-builder/api/added-tokens.mdx index 207eea88..f5adc230 100644 --- a/docs/source-doc-builder/api/added-tokens.mdx +++ b/docs/source-doc-builder/api/added-tokens.mdx @@ -5,6 +5,11 @@ ## AddedToken[[tokenizers.AddedToken]] [[autodoc]] tokenizers.AddedToken + - content + - lstrip + - normalized + - rstrip + - single_word The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. diff --git a/docs/source-doc-builder/api/encoding.mdx b/docs/source-doc-builder/api/encoding.mdx index 53b520fe..e8ab29e8 100644 --- a/docs/source-doc-builder/api/encoding.mdx +++ b/docs/source-doc-builder/api/encoding.mdx @@ -5,6 +5,18 @@ ## Encoding[[tokenizers.Encoding]] [[autodoc]] tokenizers.Encoding + - all + - attention_mask + - ids + - n_sequences + - offsets + - overflowing + - sequence_ids + - special_tokens_mask + - tokens + - type_ids + - word_ids + - words The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website. diff --git a/docs/source-doc-builder/api/tokenizer.mdx b/docs/source-doc-builder/api/tokenizer.mdx index d9dfc1d3..ab351030 100644 --- a/docs/source-doc-builder/api/tokenizer.mdx +++ b/docs/source-doc-builder/api/tokenizer.mdx @@ -5,6 +5,14 @@ ## Tokenizer[[tokenizers.Tokenizer]] [[autodoc]] tokenizers.Tokenizer + - all + - decoder + - model + - normalizer + - padding + - post_processor + - pre_tokenizer + - truncation The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.