mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
35f338a7b89e5240794fc0b75e0e43038fc401be
tokenizers/bindings/python/benches
History
Nicolas Patry 35f338a7b8 Add benchmark vs tiktoken (#1582) 
* Adding a simple tiktoken benchmark.

* Adding 1 large fused document case.
2024-07-31 17:09:23 +02:00
..
test_tiktoken.py
Add benchmark vs tiktoken (#1582)
2024-07-31 17:09:23 +02:00