mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixing broken link. (#1268)

Browse Source
This commit is contained in:
Nicolas Patry
2023-06-06 11:10:28 +02:00
committed by GitHub
parent cb819724ef
commit c7102c4c0f
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/quicktour.rst
View File

@ -73,7 +73,7 @@ Training the tokenizer
In this tour, we will build and train a Byte-Pair Encoding (BPE) tokenizer. For more information In this tour, we will build and train a Byte-Pair Encoding (BPE) tokenizer. For more information
about the different type of tokenizers, check out this `guide about the different type of tokenizers, check out this `guide
<https://huggingface.co/transformers/tokenizer_summary.html>`__ in the 🤗 Transformers <https://huggingface.co/docs/transformers/main/en/tokenizer_summary#summary-of-the-tokenizers>`__ in the 🤗 Transformers
documentation. Here, training the tokenizer means it will learn merge rules by: documentation. Here, training the tokenizer means it will learn merge rules by:
- Start with all the characters present in the training corpus as tokens. - Start with all the characters present in the training corpus as tokens.