mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 16:25:30 +00:00
11 lines
250 B
ReStructuredText
11 lines
250 B
ReStructuredText
The tokenization pipeline
|
|
====================================================================================================
|
|
|
|
TODO: Describe the tokenization pipeline:
|
|
|
|
- Normalization
|
|
- Pre-tokenization
|
|
- Tokenization
|
|
- Post-processing
|
|
- Decoding
|