* Move to maturing mimicking move for `safetensors`.
* Tmp.
* Fix sdist.
* Wat?
* Clippy 1.72
* Remove if.
* Conda sed.
* Fix doc check workflow.
* Moving to maturin AND removing http + openssl mess (smoothing transition
moving to `huggingface_hub`)
* Fix dep
* Black.
* New node bindings.
* Fix docs + node cache ?
* Yarn.
* Working dir.
* Extension module.
* Put back interpreter.
* Remove cache.
* New attempt
* Multi python.
* Remove FromPretrained.
* Remove traces of `fromPretrained`.
* Drop 3.12 for windows?
* Typo.
* Put back the default feature for ignoring links during simple test.
* Fix ?
* x86_64 -> x64.
* Remove warning for windows bindings.
* Excluse aarch.
* Include/exclude.
* Put back workflows in correct states.
* CD backports
follow
huggingface/safetensors#317
* fix node bindings?
`cargo check` doesnt work on my local configuration from `tokenizers/bindings/node/native`
i don't think it will be a problem but i have difficulty telling
* backport #315
* safetensors#317 back ports
* Split `get_n_added_tokens` into separate method
* Modify `TokenizerImpl.with_truncation()` to raise an error if given bad parameters
* Return Python error if `tokenizer.with_truncation()` fails
* Add dummy variable assignment for `no_truncation()` case
* Unrelated fmt fix.
---------
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
* fix documentation regarding regex
Split() in pre_tokenizers.rs and normalizations take a regex that is required to be built with a tokenizer specific regex module.
Clarify this in the documentation.
* Update __init__.pyi
fixed __init__.pyi
* Update bindings/python/py_src/tokenizers/__init__.pyi
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Update bindings/python/py_src/tokenizers/__init__.pyi
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Revert "Update bindings/python/py_src/tokenizers/__init__.pyi"
This reverts commit 6e8bdfcddf67bcdd8e3b1a78685fd5ef8f6a153c.
* Revert "Update bindings/python/py_src/tokenizers/__init__.pyi"
This reverts commit 897b0c0de471ad7cb6269b8456347c4e5cff2aaf.
* Revert "Update __init__.pyi"
This reverts commit fbe82310b7728ee7cdb6f8b38fbc2388f9d95771.
* add codeblocks the right way
* add codeblocks with stub.py
ran setup.py install to build, and then ran stub.py
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Makes `decode` and `decode_batch` work on borrowed content.
* Make `decode_batch` work with borrowed content.
* Fix lint.
* Attempt to map it into Node.
* Second attempt.
* Step by step.
* One more step.
* Fix lint.
* Please ...
* Removing collect.
* Revert "Removing collect."
This reverts commit 2f7ec04dc84df3cc5488625a4fcb492fdc3545e2.
---------
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
