mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,842 Commits 1 Branch 0 Tags
f0c48bd89a442819b39605ca117ecabd293bfdd7
Commit Graph

1842 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Arthur
f55822baea [pre_tokenizers] Fix sentencepiece based Metaspace (#1357) 
* nits

* allow for legacy beahaviour without making any breaking changes

* add a todo

* set to legacy by default

* skip legacy serialization

* push correct update

* lint

* add deserialization test

* add a python test as well

* updates

* fix serialization tests

* nits

* python stylijng of the tests

* better tests

* fix offsets

* fix imports

* fmt

* update metaspace

* remove TODO

* use enm

* fix some tses

* nits

* use enum

* update tests

* syling

* remove impl from for PrependScheme

* use simple getters and setters

* lint

* update tests

* add test new == new_with_prepend_scheme

* revert a change

* use setters and getterts

* Update bindings/python/src/pre_tokenizers.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* nits

* use copy rather than ref

* nits format

* more nits

* allow option string

* enforce First Never Always camel cased

* nits

* refactor

* update test as well

* fmt

* nits

* properly error out

* Update bindings/python/src/pre_tokenizers.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* suggestion changes

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
 2023-11-14 18:05:07 +01:00
Lucain
ee2af9e99a Allow huggingface_hub<1.0 (#1385) 2023-11-10 13:51:07 +01:00
Mario Šaško
648b33a09e Allow hf_hub 0.18 (#1383) 2023-11-06 14:12:05 +01:00
dependabot[bot]
c718c53bb9 Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node (#1370) 
Bumps [@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse) from 7.22.11 to 7.23.2.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse)

---
updated-dependencies:
- dependency-name: "@babel/traverse"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-25 08:14:32 +02:00
Remy
985d49ae64 fix: remove useless token (#1371) 2023-10-19 14:29:01 +02:00
天地
0d8c57da48 fix a clerical error in the comment (#1356) 2023-10-10 21:31:44 +02:00
Nicolas Patry
4322056e6e Preparing release. (#1355) 
* Preparing release.

* Fix new clippy
 2023-10-06 12:56:36 +02:00
Nicolas Patry
aed491df8c Fixing the progressbar. (#1353) 
* Fixing the progressbar.

* Upgrade deps.

* Update cargo audit

* Ssh this action.

* Fixing esaxx by using slower rust version.

* Trying the new esaxx version.

* Publish.

* Get cache again.
 2023-10-05 15:33:58 +02:00
Arthur
7e8e69a22c Let's allow hf_hub < 1.0 (#1344) 
* Let's allow hf_hub < 1.0

* Update bindings/python/pyproject.toml
 2023-10-02 14:30:10 +02:00
Anthony Platanios
18bd5e8f9d Added ability to inspect a 'Sequence' pre-tokenizer. (#1341) 
* Added ability to inspect a 'Sequence' pre-tokenizer.

* Added ability to inspect a 'Sequence' pre-tokenizer.

* Added ability to inspect a 'Sequence' pre-tokenizer.

* Linting error.

* Fix.

* Revert rename,
 2023-09-21 08:10:16 +02:00
Arthur
2c565e42c7 update package version for dev (#1339) 2023-09-07 16:19:24 +02:00
Arthur
3dce63f062 Merge pull request #1335 from ArthurZucker/update-added-tokens 
Update added tokens
 2023-09-07 12:48:54 +02:00
Arthur Zucker
efec086f35 get_added_tokens_decoder returns BTREEMap 2023-09-06 12:24:30 +00:00
Arthur Zucker
a7ace4480d python stub.py 2023-09-05 17:33:14 +00:00
Arthur Zucker
f435af8b71 linting 2023-09-05 16:43:06 +00:00
Arthur Zucker
26fdfc2bc3 style 2023-09-05 16:42:45 +00:00
Arthur Zucker
b57e1c3f5d #[allow(dead_code)] // Suppress the "method is never used" warning 2023-09-05 16:42:22 +00:00
Arthur Zucker
c3fa75fa0e nits 2023-09-05 15:40:13 +00:00
Arthur Zucker
08af8ea9c3 make tests happy 2023-09-05 15:37:09 +00:00
Arthur Zucker
531b06f6db update the get_vocab_size to compute actual length of the get_vocab function 2023-09-05 15:19:50 +00:00
Arthur Zucker
f1da83f358 add support for get_added_tokens_decoder 2023-09-05 14:49:29 +00:00
Arthur Zucker
e5fc051ad2 update 2023-09-05 13:34:43 +00:00
Arthur Zucker
93b37f36dc styling 2023-09-04 20:54:55 +00:00
Arthur Zucker
058e34b421 make special editable as well 2023-09-04 20:54:29 +00:00
Arthur Zucker
2291c89896 python stub.py 2023-09-04 19:49:36 +00:00
Arthur Zucker
b235f85527 clippy 2023-09-04 19:31:48 +00:00
Arthur Zucker
9aab096da8 fmt 2023-09-04 19:31:05 +00:00
Arthur Zucker
a59bb76aa1 update and todo 2023-09-04 19:21:38 +00:00
Arthur Zucker
c599db1421 nits 2023-09-04 19:11:19 +00:00
Arthur Zucker
d4008b0d7a cliipy 2023-09-04 19:11:05 +00:00
Arthur Zucker
b117ac7f16 updates 2023-09-04 19:10:22 +00:00
Arthur Zucker
a53dff9bc5 make content writable in python 2023-09-04 18:18:21 +00:00
Arthur Zucker
d9829cdc6e fix more tests 2023-09-04 17:22:27 +00:00
Arthur Zucker
39bd27e673 fix build 2023-09-01 21:22:07 +00:00
Arthur Zucker
9f0c703f03 update init and src for bingings python 2023-09-01 21:07:01 +00:00
Arthur Zucker
587748ab09 clean derive partial eq 2023-09-01 20:50:34 +00:00
Arthur Zucker
fdef4a118b fmt 2023-09-01 20:48:47 +00:00
Arthur Zucker
d1566a9ecc update, // AddedTokens can be updated if value changed 2023-09-01 20:48:36 +00:00
Arthur Zucker
399c6fe852 fix and update tes 2023-09-01 20:40:06 +00:00
Arthur Zucker
2b72017e17 correctly compute the new id: we take the max of the AddedToken + get_vocab_size 2023-09-01 19:03:33 +00:00
Arthur Zucker
db319492f7 clippy 2023-09-01 18:57:39 +00:00
Arthur Zucker
2dca476810 fix some tests 2023-09-01 18:48:50 +00:00
Arthur Zucker
6cca5716af fix one test? 2023-09-01 18:42:30 +00:00
Arthur Zucker
345b4eba96 updates 2023-09-01 18:41:36 +00:00
Nicolas Patry
8e522a38d9 Updating the docs with the new command. (#1333) 2023-08-29 13:15:26 +02:00
Nicolas Patry
d2010d5165 Move to maturing mimicking move for safetensors. + Rewritten node bindings. (#1331) 
* Move to maturing mimicking move for `safetensors`.

* Tmp.

* Fix sdist.

* Wat?

* Clippy 1.72

* Remove if.

* Conda sed.

* Fix doc check workflow.

* Moving to maturin AND removing http + openssl mess (smoothing transition
moving to `huggingface_hub`)

* Fix dep

* Black.

* New node bindings.

* Fix docs + node cache ?

* Yarn.

* Working dir.

* Extension module.

* Put back interpreter.

* Remove cache.

* New attempt

* Multi python.

* Remove FromPretrained.

* Remove traces of `fromPretrained`.

* Drop 3.12 for windows?

* Typo.

* Put back the default feature for ignoring links during simple test.

* Fix ?

* x86_64 -> x64.

* Remove warning for windows bindings.

* Excluse aarch.

* Include/exclude.

* Put back workflows in correct states.
 2023-08-28 16:24:14 +02:00
Nicolas Patry
f2952020d5 Python 38 arm (#1330) 2023-08-23 16:29:16 +02:00
Nicolas Patry
f08058ab2b Reduce number of different revisions by 1 (#1329) 2023-08-23 15:57:36 +02:00
Nicolas Patry
6c350d88fe Re-using scritpts from safetensors. (#1328) 2023-08-23 15:37:38 +02:00
Arthur
d0bb35d5a6 Merge pull request #1316 from boyleconnor/add-expect-for-no-truncation 
Add `expect()` for disabling truncation
 2023-08-18 19:30:53 +02:00