Add @property docs

This commit is contained in:
Mishig Davaadorj
2022-04-18 11:58:52 +02:00
parent fd005a7c4e
commit f6ba840e3e
3 changed files with 25 additions and 0 deletions

View File

@ -5,6 +5,11 @@
## AddedToken[[tokenizers.AddedToken]]
[[autodoc]] tokenizers.AddedToken
- content
- lstrip
- normalized
- rstrip
- single_word
</python>
<rust>
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.

View File

@ -5,6 +5,18 @@
## Encoding[[tokenizers.Encoding]]
[[autodoc]] tokenizers.Encoding
- all
- attention_mask
- ids
- n_sequences
- offsets
- overflowing
- sequence_ids
- special_tokens_mask
- tokens
- type_ids
- word_ids
- words
</python>
<rust>
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.

View File

@ -5,6 +5,14 @@
## Tokenizer[[tokenizers.Tokenizer]]
[[autodoc]] tokenizers.Tokenizer
- all
- decoder
- model
- normalizer
- padding
- post_processor
- pre_tokenizer
- truncation
</python>
<rust>
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.