mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-24 00:59:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Updated API Reference for AddedToken

Browse Source
This commit is contained in:
Anthony MOI
2020-10-06 18:03:46 -04:00
committed by Anthony MOI
parent 79f02bb7f0
commit ddabe130cd
6 changed files with 52 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/source/api/python_reference.inc
View File

@ -30,3 +30,11 @@ Tokenizer
.. autoclass:: tokenizers.Tokenizer
:members:
:undoc-members:
Added Tokens
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tokenizers.AddedToken
:members: