mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-09-01 23:09:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Update PreTokenizer part of the Pipeline page

Browse Source
This commit is contained in:
Anthony MOI
2020-10-28 12:28:57 -04:00
committed by Anthony MOI
parent 13a80050f0
commit 620769fd4b
5 changed files with 222 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
docs/source/entities.inc
View File

@ -30,6 +30,10 @@
:class:`~tokenizers.normalizers.Normalizer`
normalizers.Sequence
:class:`~tokenizers.normalizers.Sequence`
pre_tokenizers.Whitespace
:class:`~tokenizers.pre_tokenizers.Whitespace`
PreTokenizer
:class:`~tokenizers.pre_tokenizers.PreTokenizer`
.. entities:: rust
@ -63,6 +67,10 @@
:rust:trait:`~tokenizers::tokenizer::Normalizer`
normalizers.Sequence
:rust:struct:`~tokenizers::normalizers::utils::Sequence`
pre_tokenizers.Whitespace
:rust:struct:`~tokenizers::normalizers::whitespace::Whitespace`
PreTokenizer
:rust:trait:`~tokenizers::tokenizer::PreTokenizer`
.. entities:: node
@ -96,3 +104,7 @@
:obj:`Normalizer`
normalizers.Sequence
:obj:`Sequence`
pre_tokenizers.Whitespace
:obj:`Whitespace`
PreTokenizer
:obj:`PreTokenizer`