mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Basic layout - WIP

Browse Source
This commit is contained in:
Anthony MOI
2020-10-06 11:43:06 -04:00
committed by Anthony MOI
parent 51cd5166d9
commit 7eb0a9255e
4 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/source/pipeline.rst Normal file
View File

@ -0,0 +1,10 @@
The tokenization pipeline
====================================================================================================
TODO: Describe the tokenization pipeline:
- Normalization
- Pre-tokenization
- Tokenization
- Post-processing
- Decoding