mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-28 11:09:33 +00:00
11 lines
250 B
ReStructuredText
11 lines
250 B
ReStructuredText
The tokenization pipeline
|
|
====================================================================================================
|
|
|
|
TODO: Describe the tokenization pipeline:
|
|
|
|
- Normalization
|
|
- Pre-tokenization
|
|
- Tokenization
|
|
- Post-processing
|
|
- Decoding
|