mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 16:25:30 +00:00
* fix pylist * add comment about why we use PySequence * style * fix encode batch fast as well * Update bindings/python/src/tokenizer.rs Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com> * fix with capacity * stub :) --------- Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>