mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,856 Commits 1 Branch 0 Tags
c45aebd1029acfbe9e5dfe64e8b8441d9fae727a
Commit Graph

13 Commits

Author SHA1 Message Date
Arthur
c45aebd102 🚨 Support updating template processors (#1652) 
* current updates

* simplify

* set_item works, but `tokenizer._tokenizer.post_processor[1].single = ["$0", "</s>"]` does not !

* fix: `normalizers` deserialization and other refactoring

* fix: `pre_tokenizer` deserialization

* feat: add `__len__` implementation for `normalizer::PySequence`

* feat: add `__setitem__` impl for `normalizers::PySequence`

* feat: add `__setitem__` impl to `pre_tokenizer::PySequence`

* feat: add `__setitem__` impl to `post_processor::PySequence`

* test: add normalizer sequence setter check

* refactor: allow unused `processors::setter` macro

* test: add `__setitem__` test for processors & pretok

* refactor: `unwrap` -> `PyException::new_err()?`

* refactor: fmt

* refactor: remove unnecessary `pub`

* feat(bindings): add missing getters & setters for pretoks

* feat(bindings): add missing getters & setters for processors

* refactor(bindings): rewrite RwLock poison error msg

* refactor: remove debug print

* feat(bindings): add description as to why custom deser is needed

* feat: make post proc sequence elements mutable

* fix(binding): serialization

---------

Co-authored-by: Luc Georges <luc.sydney.georges@gmail.com>
 2025-01-28 14:58:35 +01:00
Nicolas Patry
3a6504d274 Upgrade to PyO3 0.23 (#1708) 
* Upgrade to PyO3 0.23

* Macos-12 deprecated?

* Clippy.

* Clippy auto ellision.
 2024-12-31 18:36:01 +01:00
Arthur
24d29f498d Update dev version and pyproject.toml (#1693) 
* update pyproject.toml

* update py dev version
 2024-11-27 16:01:48 +01:00
Arthur
29fef1e7aa [remove black] And use ruff (#1436) 
* nits

* Fixing deps.

* Ruff update.

* Import order matters.

* Fix.

* Revert ruff fix.

* Visualizer.

* Putting back the imports.

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
 2024-03-12 11:24:21 +01:00
Lucain
ee2af9e99a Allow huggingface_hub<1.0 (#1385) 2023-11-10 13:51:07 +01:00
Mario Šaško
648b33a09e Allow hf_hub 0.18 (#1383) 2023-11-06 14:12:05 +01:00
Arthur
7e8e69a22c Let's allow hf_hub < 1.0 (#1344) 
* Let's allow hf_hub < 1.0

* Update bindings/python/pyproject.toml
 2023-10-02 14:30:10 +02:00
Nicolas Patry
d2010d5165 Move to maturing mimicking move for safetensors. + Rewritten node bindings. (#1331) 
* Move to maturing mimicking move for `safetensors`.

* Tmp.

* Fix sdist.

* Wat?

* Clippy 1.72

* Remove if.

* Conda sed.

* Fix doc check workflow.

* Moving to maturin AND removing http + openssl mess (smoothing transition
moving to `huggingface_hub`)

* Fix dep

* Black.

* New node bindings.

* Fix docs + node cache ?

* Yarn.

* Working dir.

* Extension module.

* Put back interpreter.

* Remove cache.

* New attempt

* Multi python.

* Remove FromPretrained.

* Remove traces of `fromPretrained`.

* Drop 3.12 for windows?

* Typo.

* Put back the default feature for ignoring links during simple test.

* Fix ?

* x86_64 -> x64.

* Remove warning for windows bindings.

* Excluse aarch.

* Include/exclude.

* Put back workflows in correct states.
 2023-08-28 16:24:14 +02:00
Nicolas Patry
6113666624 Updating python formatting. (#1079) 
* Updating python formatting.

* Forgot gh action.

* Skipping isort to prevent circular imports.

* Updating stub.

* Removing `isort` (it contradicts `stub.py`).

* Fixing weird stub black/isort disagreeement.
 2022-10-05 15:29:33 +02:00
Anthony MOI
755e5f5c1e Remove support for Python 3.5 (#714) 
* Python - remove support for python 3.5

* revert ci

* revert build-wheels.sh

* Update CHANGELOG.md
 2021-05-24 17:31:01 -04:00
Patrick von Platen
dd399d2ad0 Split Pre-Tokenizer (#542) 
* start playing around

* make a first version

* refactor

* apply make format

* add python bindings

* add some python binding tests

* correct pre-tokenizers

* update auto-generated bindings

* lint python bindings

* add code node

* add split to docs

* refactor python binding a bit

* cargo fmt

* clippy and fmt in node

* quick updates and fixes

* Oops

* Update node typings

* Update changelog

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
 2020-11-27 17:07:03 -05:00
Anthony MOI
a42e13a644 Setup black format in pyproject.toml 2020-09-23 11:58:35 -04:00
Anthony MOI
428890d6e0 Basic python setuptools 2019-11-27 12:21:37 -05:00