mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-03 19:28:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Aho corasick version for many added tokens. (#871)

Browse Source 
* Aho corasick version.

* Remove test file.

* Compile on `stable`.
This commit is contained in:
Nicolas Patry
2022-01-06 16:04:51 +01:00
committed by GitHub
parent fb837b4adb
commit 076319d542
6 changed files with 98 additions and 133 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
bindings/python/Cargo.lock generated
View File

@@ -1745,6 +1745,7 @@ checksum = "cda74da7e1a664f795bb1f8a87ec406fb89a02522cf6e50620d016add6dbbf5c"
name = "tokenizers"
version = "0.11.0"
dependencies = [
"aho-corasick",
"cached-path",
"clap",
"derive_builder",