mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,788 Commits 1 Branch 0 Tags
1df498a186bed26ded6c638547dd8359d836bb21
Commit Graph

3 Commits

Author SHA1 Message Date
Nicolas Patry
1df498a186 Fixing benchmark2. 2024-08-01 15:52:39 +02:00
Nicolas Patry
c6f2c0b057 Fixing the benchmark. (#1583) 2024-08-01 10:36:53 +02:00
Nicolas Patry
35f338a7b8 Add benchmark vs tiktoken (#1582) 
* Adding a simple tiktoken benchmark.

* Adding 1 large fused document case.
 2024-07-31 17:09:23 +02:00