Commit Graph

1022 Commits

Author SHA1 Message Date
86138337fc Using serde (serde_pyo3) to get __str__ and __repr__ easily. 2024-08-02 18:41:54 +02:00
7415e28536 Enabling the option to use fancy_regex instead of onig. 2024-08-01 15:53:17 +02:00
1df498a186 Fixing benchmark2. 2024-08-01 15:52:39 +02:00
c6f2c0b057 Fixing the benchmark. (#1583) 2024-08-01 10:36:53 +02:00
35f338a7b8 Add benchmark vs tiktoken (#1582)
* Adding a simple tiktoken benchmark.

* Adding 1 large fused document case.
2024-07-31 17:09:23 +02:00
4ea2f235b0 Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555)
* feature dependent test

* nit about 嗎

* update

* actuallyfix it

* update the test

add it

fix

* stub

* Update tokenizers/src/pre_tokenizers/byte_level.rs

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>

* skip failing test

* add normalizer to init

---------

Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
2024-07-15 12:12:03 +02:00
fdd26ba9a3 Enable dropout = 0.0 as an equivalent to none in BPE (#1550)
* enable dropout = 0.0

* typo

* lint

* formatter

* enable dropout = 0.0

* formatter
2024-06-24 12:36:11 +02:00
1ff56c0c70 Fix 'dictionnary' typo (#1511) 2024-06-11 15:43:47 +02:00
88f51fe7d2 Switch from cached_download to hf_hub_download in tests (#1547) 2024-06-11 15:26:58 +02:00
f2ec3b239b remove enforcement of non special when adding tokens (#1521)
* remove enforcement of non special when adding tokens

* mut no longer needed

* add a small test

* nit

* style

* audit

* ignore cargo audit's own vulnerability

* update

* revert

* remove CVE
2024-04-30 15:53:47 +02:00
71c2a8d01a update dev version so 0.19.1 2024-04-17 23:17:12 +02:00
91393ef75e Fixing doc. (#1499)
* Fixing doc.

* SentencePieceUnigram  and Convert.py still used sentencepiece

* stub

---------

Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
2024-04-17 09:32:40 +02:00
949d9e3e0e Bumping all versions 3 times (ty transformers :) ) (#1498) 2024-04-16 15:58:36 +02:00
d5a8cc7a49 PyO3 0.21. (#1494)
* PyO3 0.21.

* Upgraded everything.

* Rustfmt.
2024-04-16 13:49:52 +02:00
914576f7ed Add more support for tiktoken based tokenizers (#1493)
* first commit

* update

* clippy

* lint

* clippy and lint

* fmt

* revert print

* 😈

* style

* add a test

* more fmt

* Use ignore_merges

* stub

* fix

* update

* Update tokenizers/src/models/bpe/model.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* update

* rust lint

* dob; t repeat yourself

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2024-04-15 17:26:36 +02:00
6e58f838b3 version = "0.16.0-dev.0" 2024-04-02 09:51:14 +02:00
09069717e9 Refactor metaspace (#1476)
* version = "0.15.3-dev-0”

Improve performances of meta space, but also just fix it.

(transformers) ➜  transformers git:(refactor-default-llama) ✗ python ../scripts/gemma-dummy.py
Token indices sequence length is longer than the specified maximum sequence length for this model (14999 > 2048). Running this sequence through the model will result in indexing errors
['<REPR_END>', '▁inform', '<s>', '.', '▁Hey', '<unk>', '.', '▁', '▁', '▁', '▁', '▁', '▁', '▁.']
['▁inform', '<s>', '.', '▁Hey', '<unk>', '.', '▁', '▁', '▁', '▁', '▁', '▁', '▁.']
[0.0006330013275146484, 0.0014591217041015625, 0.015890836715698242, 0.18584918975830078, 2.1726326942443848]
(transformers) ➜  transformers git:(refactor-default-llama) ✗ python ../scripts/gemma-dummy.py
Token indices sequence length is longer than the specified maximum sequence length for this model (10000 > 2048). Running this sequence through the model will result in indexing errors
['<REPR_END>', 'in', 'form', '<s>', '.', '▁Hey', '<unk>', '.', '▁▁▁▁▁▁', '▁.']
['in', 'form', '<s>', '.', '▁Hey', '<unk>', '.', '▁▁▁▁▁▁', '▁.']
[0.0008409023284912109, 0.0008909702301025391, 0.00882411003112793, 0.10214710235595703, 1.187899112701416]

* well what do we have

* nit

* be BC with non legacy

* unrelated change for clippy

* fix test

* splitting is a must for word_ids

* fmt and lint

* Fixing everything (hopefully better).

* Fixing node.

* Including yarn.lock

* Lint.

* Stubs.

* revert to use split

* fix merge issues

* fix tests

* finish fixing tests

* ruff

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2024-03-30 10:27:24 +01:00
d8c4388166 Bump ip from 2.0.0 to 2.0.1 in /bindings/node (#1456)
Bumps [ip](https://github.com/indutny/node-ip) from 2.0.0 to 2.0.1.
- [Commits](https://github.com/indutny/node-ip/compare/v2.0.0...v2.0.1)

---
updated-dependencies:
- dependency-name: ip
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-25 11:29:36 +01:00
29fef1e7aa [remove black] And use ruff (#1436)
* nits

* Fixing deps.

* Ruff update.

* Import order matters.

* Fix.

* Revert ruff fix.

* Visualizer.

* Putting back the imports.

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2024-03-12 11:24:21 +01:00
7f49f20ab0 version = "0.15.3-dev-0” 2024-02-12 09:48:00 +09:00
4a8105c366 Convert word counts to u64 (#1433)
* Convert word counts to u64

* More spots needed to compile
2024-02-06 03:39:12 +01:00
8f73fe9515 update dev version to 0.15.2-dev.0 2024-01-22 15:34:57 +01:00
6a77d4859b Encode special tokens (#1437)
* add doc in the code

* add option to skip special tokens

* nits

* add api dummy for now

* Fmt.

* Fix fmt.

* Fix the stub.

* add a test

* add a test in python

* style it

* nits

* add getter and setters

* stub

* update python test

* fmt

* last nit

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2024-01-19 12:43:43 +01:00
888dd4bc65 pyo3: update to 0.20 (#1386)
Co-authored-by: Mike Lui <mikelui@meta.com>
2024-01-11 17:03:13 +01:00
11462596d1 Faster HF dataset iteration in docs (#1414)
* Faster HF dataset iteration in docs

* Nit
2023-12-14 16:12:56 +01:00
e3bcef288b udpate to version = "0.15.1-dev0" (#1390)
* Apply suggestions from code review
2023-11-15 13:30:58 +01:00
f55822baea [pre_tokenizers] Fix sentencepiece based Metaspace (#1357)
* nits

* allow for legacy beahaviour without making any breaking changes

* add a todo

* set to legacy by default

* skip legacy serialization

* push correct update

* lint

* add deserialization test

* add a python test as well

* updates

* fix serialization tests

* nits

* python stylijng of the tests

* better tests

* fix offsets

* fix imports

* fmt

* update metaspace

* remove TODO

* use enm

* fix some tses

* nits

* use enum

* update tests

* syling

* remove impl from for PrependScheme

* use simple getters and setters

* lint

* update tests

* add test new == new_with_prepend_scheme

* revert a change

* use setters and getterts

* Update bindings/python/src/pre_tokenizers.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* nits

* use copy rather than ref

* nits format

* more nits

* allow option string

* enforce First Never Always camel cased

* nits

* refactor

* update test as well

* fmt

* nits

* properly error out

* Update bindings/python/src/pre_tokenizers.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* suggestion changes

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2023-11-14 18:05:07 +01:00
ee2af9e99a Allow huggingface_hub<1.0 (#1385) 2023-11-10 13:51:07 +01:00
648b33a09e Allow hf_hub 0.18 (#1383) 2023-11-06 14:12:05 +01:00
c718c53bb9 Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node (#1370)
Bumps [@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse) from 7.22.11 to 7.23.2.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse)

---
updated-dependencies:
- dependency-name: "@babel/traverse"
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-25 08:14:32 +02:00
4322056e6e Preparing release. (#1355)
* Preparing release.

* Fix new clippy
2023-10-06 12:56:36 +02:00
aed491df8c Fixing the progressbar. (#1353)
* Fixing the progressbar.

* Upgrade deps.

* Update cargo audit

* Ssh this action.

* Fixing esaxx by using slower rust version.

* Trying the new esaxx version.

* Publish.

* Get cache again.
2023-10-05 15:33:58 +02:00
7e8e69a22c Let's allow hf_hub < 1.0 (#1344)
* Let's allow hf_hub < 1.0

* Update bindings/python/pyproject.toml
2023-10-02 14:30:10 +02:00
2c565e42c7 update package version for dev (#1339) 2023-09-07 16:19:24 +02:00
efec086f35 get_added_tokens_decoder returns BTREEMap 2023-09-06 12:24:30 +00:00
a7ace4480d python stub.py 2023-09-05 17:33:14 +00:00
f435af8b71 linting 2023-09-05 16:43:06 +00:00
26fdfc2bc3 style 2023-09-05 16:42:45 +00:00
c3fa75fa0e nits 2023-09-05 15:40:13 +00:00
08af8ea9c3 make tests happy 2023-09-05 15:37:09 +00:00
f1da83f358 add support for get_added_tokens_decoder 2023-09-05 14:49:29 +00:00
93b37f36dc styling 2023-09-04 20:54:55 +00:00
058e34b421 make special editable as well 2023-09-04 20:54:29 +00:00
2291c89896 python stub.py 2023-09-04 19:49:36 +00:00
c599db1421 nits 2023-09-04 19:11:19 +00:00
d4008b0d7a cliipy 2023-09-04 19:11:05 +00:00
b117ac7f16 updates 2023-09-04 19:10:22 +00:00
a53dff9bc5 make content writable in python 2023-09-04 18:18:21 +00:00
d9829cdc6e fix more tests 2023-09-04 17:22:27 +00:00
39bd27e673 fix build 2023-09-01 21:22:07 +00:00