mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Basic layout - WIP

Browse Source
This commit is contained in:
Anthony MOI
2020-10-06 11:43:06 -04:00
committed by Anthony MOI
parent 51cd5166d9
commit 7eb0a9255e
4 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/source/index.rst
View File

@ -27,7 +27,11 @@ Components:
.. toctree::
:maxdepth: 2
:caption: Getting Started
quicktour
installation
pipeline
components
Load an existing tokenizer:

5
docs/source/installation.rst Normal file
View File

@ -0,0 +1,5 @@
Installation
====================================================================================================
- How to install using pip
- How to build from source

10
docs/source/pipeline.rst Normal file
View File

@ -0,0 +1,10 @@
The tokenization pipeline
====================================================================================================
TODO: Describe the tokenization pipeline:
- Normalization
- Pre-tokenization
- Tokenization
- Post-processing
- Decoding

4
docs/source/quicktour.rst Normal file
View File

@ -0,0 +1,4 @@
Quicktour
====================================================================================================
- How to use a tokenizer: encode, encode_batch, ``Encoding``, offsets, mappings, ...