fb292d1eae
0.13.4.rc1 ( #1319 )
2023-08-14 12:06:43 +02:00
864135bef1
Add unigram bytefallback ( #1217 )
...
* current updates will go red
* cargo fmt
* npm install
* refactor train for unigram to allow bytefallbakc (breaking)
* fmt
* nits
* update
* add a proper test
* fix encode optimised fallback + add trainer arg
* fixes
* fixes
* fix tests
* add test
* fmt
* fix rust test
* update python bindings
* update
* pub is okay and needed
* more fix
* cleanup
* remove useles id
* MissingUnkId error
* nits
* fix offset
* add a test in python
* update src bindings
* remove bytefallback from trainer
* styling
* update pckg
* lint
* fmt
* stup with dev
* update code based on review
* remove unused function
* udpate python test to compare ids
* fix option bool issues
* final fix
* clippy
* fix npm isntall
* update
* update test
* more in depth testing
* Lint
* last attempt to fix node
* update node bindings
* fmt
* Update tokenizers/src/models/unigram/model.rs
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com >
* update based on review
* simpler test
* lint
---------
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com >
2023-06-26 10:46:59 +02:00
1cb44bd180
New version 0.13.3
2023-04-04 14:14:17 +02:00
96a9e5715c
New version. ( #1082 )
...
* New version.
The actual release will happen *before* PyO3 0.17.2 because
the tests were ran before than.
* Manylinux2014 necessary now with Rust 1.64.
2022-10-06 15:45:56 +02:00
6e5569a540
Moving versions numbers to dev
mode. ( #1067 )
2022-09-22 18:24:07 +02:00
7bfab48979
Preparing rc1 release. ( #1056 )
...
* Preparing rc1 release.
* Fixing test_alignment_methods
* Fixing the overflowing sequence_id issue (LayoutLMv2 tests caught this).
* Adding overly complex overflowing test.
2022-09-12 16:07:06 +02:00
8a9bb28f46
Preparing for 0.12.1 ( #978 )
...
* Preparing for 0.12.1
* Updated the changelog.
2022-04-12 17:57:33 +02:00
0eb7455fe5
Preparing 0.12
release. ( #967 )
...
* Preparing `0.12` release.
* Fix click version: https://github.com/psf/black/issues/2964
2022-03-31 11:06:33 +02:00
a4a68de98a
Workarounds publishing issues:
...
- Upgrade package-lock.json (cannot find VS code attempt)
- Use published `macro_rules_attribute` so `cargo publish` works.
2022-02-28 11:16:46 +01:00
ffaee13994
Preparing for 0.11.6 release.
2022-02-28 10:20:49 +01:00
9b85424520
Version bump.
2022-01-17 22:30:25 +01:00
ab9a2f3100
Update versions.
2022-01-17 09:40:01 +01:00
c1100ec542
Clippy fixes. ( #846 )
...
* Clippy fixes.
* Drop support for Python 3.6
* Remove other 3.6
* Re-enabling caches for build (5h + seems too long and issue seems
solved)
https://github.com/actions/virtual-environments/issues/572
* `npm audit fix`.
* Fix yaml ?
* Pyarrow issue fixed: https://github.com/huggingface/datasets/pull/2268
* Installing dev libraries.
* Install python dev elsewhere ?
* Typo.
* No sudo.
* ...
* Testing the GH again.
* Maybe v2 will fix ?
* Fixing tests on MacOS Python 3.8+
2021-12-15 15:55:48 +01:00
884bfb7970
Prepare node release ( #794 )
...
* Node - Update changelog for release
* Update node release to add v14 & v15
Co-authored-by: Huan (李卓桓) <zixia@zixia.net >
* Node - Update version number
* Node - Update dependencies
* Node - Lint
Co-authored-by: Huan (李卓桓) <zixia@zixia.net >
2021-09-02 09:58:01 -04:00
44e8f4be8f
Fixing node.js example.
...
- Now supports more lenient syntax and more aligned with python&Rust.
- Backward compatible.
2020-11-02 17:07:27 -05:00
6f8892e3ae
Upgrade neon version + tests in JS instead of TS.
2020-11-02 17:07:27 -05:00
e108d8fcca
Node - Make sure we run all tests
2020-09-10 14:00:53 -04:00
85800e4189
Node - Update dev dependencies / fix lodash vulnerability
2020-07-24 15:43:54 -04:00
9294db78a4
Node - Version 0.7.0
2020-07-01 17:48:23 -04:00
c164baf539
Node - Version 0.6.2
2020-04-13 16:57:44 -04:00
cca20f4912
Node - Version 0.6.1
2020-04-01 14:26:39 -04:00
bb920af9e8
Node - Version 0.6.0
2020-03-30 12:38:15 -04:00
bb6efeb04f
node: version 0.5.0
2020-02-27 16:03:39 -05:00
5e612669bb
node: version 0.4.1
2020-02-11 09:54:17 -05:00
be67d51185
node: add more infos in package.json
2020-02-05 18:07:39 -05:00
3df188dc27
node: version 0.4.0
2020-02-05 17:38:59 -05:00
2bcd47440c
node: add enums for padding and truncation strategies
2020-02-05 14:28:53 -05:00
3f6de4d33f
node: add lint check in workflow
2020-01-29 16:58:59 -05:00
d977d82a72
node: update ts build
2020-01-29 11:17:48 -05:00
88391dd185
node: bump version to 0.3.1
2020-01-24 17:20:09 -05:00
0e724dfeb4
node: remove redundant types in jsdoc
2020-01-24 17:20:05 -05:00
c7f850415f
generalize npm cache and forget about rust target
...
neon re-triggers complete rust build every time because of `artifacts.json` which is generated every time... (and cannot be versioned since it varies by platform)
2020-01-23 17:27:14 -05:00
68cce806cd
Node bindings v0.3.0
...
- Fix inconsistencies in BPETokenizer with the addition of a `whitespaceSplit` pre-tokenizer
- Fix a bug with the IDs associated with added tokens.
2020-01-22 18:24:48 -05:00
2aef3e9d9c
add eslint/prettier
2020-01-22 18:08:22 -05:00
6b8deb90e2
prepare for tests
2020-01-22 17:37:30 -05:00
a779714a9e
don't forget to copy README!!!
2020-01-14 17:31:29 -05:00
25546d00c6
bump node bindings version: 0.2.3
2020-01-14 17:08:57 -05:00
4d70d0b1c2
complete windows build
...
fix tsc
test reading package version
add working dir...
test upload tarball
let's add some python
test ubuntu build
test windows build
get rid of package env var
fix working dir
2020-01-14 15:20:34 -05:00
f836f2109b
build ts
2020-01-14 15:20:34 -05:00
df67eadeca
fix path bin
2020-01-10 15:53:56 -05:00
c9da0ffa18
bump
2020-01-10 15:35:27 -05:00
e68b4ae501
publish script
2020-01-10 15:19:59 -05:00
66d65595f6
clean package / package-lock
2020-01-10 11:53:30 -05:00
6b0935d5de
first implementations draft
2020-01-10 11:53:30 -05:00
0b8a51c010
First draft node typings
2020-01-10 11:53:30 -05:00
57a1ce7e1d
Node bindings backbone
2019-11-01 16:39:03 -04:00