mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Update API Reference for Encoding

Browse Source
This commit is contained in:
Anthony MOI
2020-10-09 12:37:21 -04:00
committed by Anthony MOI
parent 12af3f2240
commit d2fc0e4836
3 changed files with 282 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/source/api/python.inc
View File

@ -30,7 +30,13 @@ Tokenizer
.. autoclass:: tokenizers.Tokenizer
:members:
:undoc-members:
Encoding
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. autoclass:: tokenizers.Encoding
:members:
Added Tokens