mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-04 19:58:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixed Dead Link: Build your own #435 (#436)

Browse Source 
* Fixed Dead Link: Build your own #435

* Update bindings/python/README.md

Co-authored-by: Anthony MOI <xn1t0x@gmail.com>
This commit is contained in:
Dagmawi Moges
2020-09-25 16:41:31 +03:00
committed by GitHub
parent a0a163fd62
commit 7f8b357b92
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/README.md
View File

@@ -118,7 +118,7 @@ All of these can be used and trained as explained above!
Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer,
by putting all the different parts you need together. by putting all the different parts you need together.
You can how we implemented the [provided tokenizers](https://github.com/huggingface/tokenizers/tree/master/bindings/python/tokenizers/implementations) and adapt them easily to your own needs. You can check how we implemented the [provided tokenizers](https://github.com/huggingface/tokenizers/tree/master/bindings/python/py_src/tokenizers/implementations) and adapt them easily to your own needs.
#### Building a byte-level BPE #### Building a byte-level BPE