Nicolas Patry bfd9cdeefb Perf improvement 16% by removing offsets. (#1587) ...

* [Breaking Change] Perf improvement 16% by removing offsets.

Offsets calculation are always calculated in Python land.
By changing it to not being calculated, we win 16% of the runtime.

This is not the total extent of it because offsets are
still calculated in bytes.

* Required features.

* Remove clippy error.

* Make it non breaking and still show perf improvement.

* Even faster without offsets.

* Update doc.

* Fmt.

* Apply suggestions from code review

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* fmt.

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

2024-08-08 14:56:13 +02:00

..

test_tiktoken.py

Perf improvement 16% by removing offsets. (#1587)

2024-08-08 14:56:13 +02:00