mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 16:25:30 +00:00
152880ab3e
* Add truncation to enable_truncation * Fix typo * Adding truncation_side within `TruncationParams`. * Node serialization of this direction param. * Update the test. * Fixing warnings/lint. * Adding stuff (can't local debug :( ) * Slow loop... ;( * Stub.py. Co-authored-by: Niels Rogge <niels.rogge1@gmail.com>
37 lines
1.6 KiB
Plaintext
37 lines
1.6 KiB
Plaintext
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|
|
<DOCUMENT> \test{bla} thisisatest </DOCUMENT>
|