mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Update Rust entities using roles

Browse Source
This commit is contained in:
Anthony MOI
2020-10-21 18:50:11 -04:00
committed by Anthony MOI
parent 4510712295
commit bd84240a20
2 changed files with 7 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/source/index.rst
View File

@ -65,6 +65,10 @@ Main features:
classmethod
static method
Tokenizer
`Tokenizer <https://docs.rs/tokenizers/latest/tokenizers/tokenizer/struct.Tokenizer.html>`__
:rust:struct:`~tokenizers::tokenizer::Tokenizer`
Tokenizer.train
`train <https://docs.rs/tokenizers/0.10.1/tokenizers/tokenizer/struct.Tokenizer.html#method.train>`__
:rust:meth:`~tokenizers::tokenizer::Tokenizer::train()`
Tokenizer.save
:rust:meth:`~tokenizers::tokenizer::Tokenizer::save()`
Tokenizer.from_file
:rust:meth:`~tokenizers::tokenizer::Tokenizer::from_file()`

2
docs/source/quicktour.rst
View File

@ -38,7 +38,7 @@ Training the tokenizer
.. entities:: rust
BpeTrainer
`BpeTrainer <https://docs.rs/tokenizers/latest/tokenizers/models/bpe/struct.BpeTrainer.html>`__
:rust:struct:`~tokenizers::models::bpe::BpeTrainer`
vocab_size
:obj:`vocab_size`
min_frequency