Files
tokenizers/docs/source-doc-builder/api/tokenizer.mdx
Mishig Davaadorj 6eda286ab1 Init new docs
2022-04-18 09:37:14 +02:00

15 lines
341 B
Plaintext

# Tokenizer
<tokenizerslangcontent>
<python>
## Tokenizer[[tokenizers.Tokenizer]]
[[autodoc]] tokenizers.Tokenizer
</python>
<rust>
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.
</rust>
<node>
The node API has not been documented yet.
</node>
</tokenizerslangcontent>