mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,787 Commits 1 Branch 0 Tags
c6f2c0b05714d745b4b83153b34df09a817a4559
Commit Graph

2 Commits

Author SHA1 Message Date
Nicolas Patry
c6f2c0b057 Fixing the benchmark. (#1583) 2024-08-01 10:36:53 +02:00
Nicolas Patry
35f338a7b8 Add benchmark vs tiktoken (#1582) 
* Adding a simple tiktoken benchmark.

* Adding 1 large fused document case.
 2024-07-31 17:09:23 +02:00