mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Doc - Improve snippets testing

Browse Source
This commit is contained in:
Anthony MOI
2020-10-02 15:52:33 -04:00
committed by Anthony MOI
parent f4e7754112
commit 000c19a7a5
12 changed files with 84 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/source/index.rst
View File

@ -50,10 +50,11 @@ Loading a previously saved tokenizer is extremely simple and requires a single l
.. only:: Python
.. literalinclude:: ../../bindings/python/tests/examples/test_load.py
.. literalinclude:: ../../bindings/python/tests/documentation/test_load.py
:language: python
:start-after: START load
:end-before: END load
:start-after: START load_tokenizer
:end-before: END load_tokenizer
:dedent: 4
.. only:: Node
@ -79,10 +80,16 @@ Small guide of :ref:`how to create a Tokenizer options<tokenizer_blocks>`.
.. only:: Python
.. literalinclude:: ../../bindings/python/tests/examples/test_train.py
.. literalinclude:: ../../bindings/python/tests/documentation/test_train.py
:language: python
:start-after: START train_tokenizer
:end-before: END train_tokenizer
:dedent: 4
.. only:: Node
.. literalinclude:: ../../bindings/node/examples/train.test.js
:language: javascript
:start-after: START train_tokenizer
:end-before: END train_tokenizer
:dedent: 4