mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix no-onig no-wasm builds (#1772)

Browse Source 
Now, you can build with:

```
--no-default-features --features=fancy-regex
```

Which previously didn't work. You had to enable the `unstable_wasm`
flag.

I think using `fancy_regex` without wasm is a valid use-case, as I've
seen extremely slow build times using `onig`.
See: https://github.com/huggingface/tokenizers/issues/1730

Onig also breaks, sometimes, with compiler updates.
See: https://github.com/huggingface/tokenizers/pull/1771

Closes https://github.com/huggingface/tokenizers/issues/1729
This commit is contained in:
Owen Shepherd
2025-05-27 06:44:20 +01:00
committed by GitHub
parent 759d7aa77a
commit 67db0cd1dd
1 changed files with 10 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
tokenizers/src/utils/mod.rs
View File

@ -2,15 +2,21 @@ pub(crate) mod cache;
#[cfg(feature = "http")]
pub(crate) mod from_pretrained;
#[cfg(feature = "unstable_wasm")]
#[cfg(feature = "fancy-regex")]
mod fancy;
#[cfg(feature = "unstable_wasm")]
#[cfg(feature = "fancy-regex")]
pub use fancy::SysRegex;
#[cfg(not(feature = "unstable_wasm"))]
#[cfg(feature = "onig")]
mod onig;
#[cfg(not(feature = "unstable_wasm"))]
#[cfg(feature = "onig")]
pub use crate::utils::onig::SysRegex;
#[cfg(all(feature = "onig", feature = "fancy-regex"))]
compile_error!("Features `onig` and `fancy-regex` are mutually exclusive");
#[cfg(not(any(feature = "onig", feature = "fancy-regex")))]
compile_error!("One of the `onig`, or `fancy-regex` features must be enabled");
pub mod iter;
pub mod padding;
pub mod parallelism;