Commit Graph

46 Commits

Author SHA1 Message Date
fb292d1eae 0.13.4.rc1 (#1319) 2023-08-14 12:06:43 +02:00
864135bef1 Add unigram bytefallback (#1217)
* current updates will go red

* cargo fmt

* npm install

* refactor train for unigram to allow bytefallbakc (breaking)

* fmt

* nits

* update

* add a proper test

* fix encode optimised fallback + add trainer arg

* fixes

* fixes

* fix tests

* add test

* fmt

* fix rust test

* update python bindings

* update

* pub is okay and needed

* more fix

* cleanup

* remove useles id

* MissingUnkId error

* nits

* fix offset

* add a test in python

* update src bindings

* remove bytefallback from trainer

* styling

* update pckg

* lint

* fmt

* stup with dev

* update code based on review

* remove unused function

* udpate python test to compare ids

* fix option bool issues

* final fix

* clippy

* fix npm isntall

* update

* update test

* more in depth testing

* Lint

* last attempt to fix node

* update node bindings

* fmt

* Update tokenizers/src/models/unigram/model.rs

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

* update based on review

* simpler test

* lint

---------

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
2023-06-26 10:46:59 +02:00
1cb44bd180 New version 0.13.3 2023-04-04 14:14:17 +02:00
96a9e5715c New version. (#1082)
* New version.

The actual release will happen *before* PyO3 0.17.2 because
the tests were ran before than.

* Manylinux2014 necessary now with Rust 1.64.
2022-10-06 15:45:56 +02:00
6e5569a540 Moving versions numbers to dev mode. (#1067) 2022-09-22 18:24:07 +02:00
7bfab48979 Preparing rc1 release. (#1056)
* Preparing rc1 release.

* Fixing test_alignment_methods

* Fixing the overflowing sequence_id issue (LayoutLMv2 tests caught this).

* Adding overly complex overflowing test.
2022-09-12 16:07:06 +02:00
8a9bb28f46 Preparing for 0.12.1 (#978)
* Preparing for 0.12.1

* Updated the changelog.
2022-04-12 17:57:33 +02:00
0eb7455fe5 Preparing 0.12 release. (#967)
* Preparing `0.12` release.

* Fix click version: https://github.com/psf/black/issues/2964
2022-03-31 11:06:33 +02:00
a4a68de98a Workarounds publishing issues:
- Upgrade package-lock.json (cannot find VS code attempt)
- Use published `macro_rules_attribute` so `cargo publish` works.
2022-02-28 11:16:46 +01:00
ffaee13994 Preparing for 0.11.6 release. 2022-02-28 10:20:49 +01:00
9b85424520 Version bump. 2022-01-17 22:30:25 +01:00
ab9a2f3100 Update versions. 2022-01-17 09:40:01 +01:00
c1100ec542 Clippy fixes. (#846)
* Clippy fixes.

* Drop support for Python 3.6

* Remove other 3.6

* Re-enabling caches for build (5h + seems too long and issue seems
solved)

https://github.com/actions/virtual-environments/issues/572

* `npm audit fix`.

* Fix yaml ?

* Pyarrow issue fixed: https://github.com/huggingface/datasets/pull/2268

* Installing dev libraries.

* Install python dev elsewhere ?

* Typo.

* No sudo.

* ...

* Testing the GH again.

* Maybe v2 will fix ?

* Fixing tests on MacOS Python 3.8+
2021-12-15 15:55:48 +01:00
884bfb7970 Prepare node release (#794)
* Node - Update changelog for release

* Update node release to add v14 & v15

Co-authored-by: Huan (李卓桓) <zixia@zixia.net>

* Node - Update version number

* Node - Update dependencies

* Node - Lint

Co-authored-by: Huan (李卓桓) <zixia@zixia.net>
2021-09-02 09:58:01 -04:00
44e8f4be8f Fixing node.js example.
- Now supports more lenient syntax and more aligned with python&Rust.
- Backward compatible.
2020-11-02 17:07:27 -05:00
6f8892e3ae Upgrade neon version + tests in JS instead of TS. 2020-11-02 17:07:27 -05:00
e108d8fcca Node - Make sure we run all tests 2020-09-10 14:00:53 -04:00
85800e4189 Node - Update dev dependencies / fix lodash vulnerability 2020-07-24 15:43:54 -04:00
9294db78a4 Node - Version 0.7.0 2020-07-01 17:48:23 -04:00
c164baf539 Node - Version 0.6.2 2020-04-13 16:57:44 -04:00
cca20f4912 Node - Version 0.6.1 2020-04-01 14:26:39 -04:00
bb920af9e8 Node - Version 0.6.0 2020-03-30 12:38:15 -04:00
bb6efeb04f node: version 0.5.0 2020-02-27 16:03:39 -05:00
5e612669bb node: version 0.4.1 2020-02-11 09:54:17 -05:00
be67d51185 node: add more infos in package.json 2020-02-05 18:07:39 -05:00
3df188dc27 node: version 0.4.0 2020-02-05 17:38:59 -05:00
2bcd47440c node: add enums for padding and truncation strategies 2020-02-05 14:28:53 -05:00
3f6de4d33f node: add lint check in workflow 2020-01-29 16:58:59 -05:00
d977d82a72 node: update ts build 2020-01-29 11:17:48 -05:00
88391dd185 node: bump version to 0.3.1 2020-01-24 17:20:09 -05:00
0e724dfeb4 node: remove redundant types in jsdoc 2020-01-24 17:20:05 -05:00
c7f850415f generalize npm cache and forget about rust target
neon re-triggers complete rust build every time because of `artifacts.json` which is generated every time... (and cannot be versioned since it varies by platform)
2020-01-23 17:27:14 -05:00
68cce806cd Node bindings v0.3.0
- Fix inconsistencies in BPETokenizer with the addition of a `whitespaceSplit` pre-tokenizer
- Fix a bug with the IDs associated with added tokens.
2020-01-22 18:24:48 -05:00
2aef3e9d9c add eslint/prettier 2020-01-22 18:08:22 -05:00
6b8deb90e2 prepare for tests 2020-01-22 17:37:30 -05:00
a779714a9e don't forget to copy README!!! 2020-01-14 17:31:29 -05:00
25546d00c6 bump node bindings version: 0.2.3 2020-01-14 17:08:57 -05:00
4d70d0b1c2 complete windows build
fix tsc

test reading package version

add working dir...

test upload tarball

let's add some python

test ubuntu build

test windows build

get rid of package env var

fix working dir
2020-01-14 15:20:34 -05:00
f836f2109b build ts 2020-01-14 15:20:34 -05:00
df67eadeca fix path bin 2020-01-10 15:53:56 -05:00
c9da0ffa18 bump 2020-01-10 15:35:27 -05:00
e68b4ae501 publish script 2020-01-10 15:19:59 -05:00
66d65595f6 clean package / package-lock 2020-01-10 11:53:30 -05:00
6b0935d5de first implementations draft 2020-01-10 11:53:30 -05:00
0b8a51c010 First draft node typings 2020-01-10 11:53:30 -05:00
57a1ce7e1d Node bindings backbone 2019-11-01 16:39:03 -04:00