Doc - Updated API Reference for AddedToken

2025-12-16 17:18:43 +00:00 · 2020-10-06 18:03:46 -04:00
parent 79f02bb7f0
commit ddabe130cd
6 changed files with 52 additions and 3 deletions
--- a/docs/source/api/node_reference.inc
+++ b/docs/source/api/node_reference.inc
@@ -1 +1,4 @@
+Documentation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 The node API has not been documented yet.
--- a/docs/source/api/python_reference.inc
+++ b/docs/source/api/python_reference.inc
@@ -30,3 +30,11 @@ Tokenizer

 .. autoclass:: tokenizers.Tokenizer
    :members:
+    :undoc-members:
+
+
+Added Tokens
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. autoclass:: tokenizers.AddedToken
+    :members:
--- a/docs/source/api/reference.rst
+++ b/docs/source/api/reference.rst
@@ -1,6 +1,3 @@
-API Reference
-====================================================================================================
-
 .. only:: python

    .. include:: python_reference.inc
--- a/docs/source/api/rust_reference.inc
+++ b/docs/source/api/rust_reference.inc
@@ -1 +1,4 @@
+Documentation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
 The Rust API has not been documented yet.