mirror of
https://github.com/mii443/tokenizers.git
synced 2025-08-22 16:25:30 +00:00
Preparing rc1 release. (#1056)
* Preparing rc1 release. * Fixing test_alignment_methods * Fixing the overflowing sequence_id issue (LayoutLMv2 tests caught this). * Adding overly complex overflowing test.
This commit is contained in:
@ -1,3 +1,8 @@
|
||||
## [0.13.0]
|
||||
|
||||
- [#1008] `Decoder` is now a composable trait, but without being backward incompatible
|
||||
- [#1047, #1051, #1052] `Processor` is now a composable trait, but without being backward incompatible
|
||||
|
||||
## [0.12.1]
|
||||
|
||||
- [#938] **Reverted breaking change**. https://github.com/huggingface/transformers/issues/16520
|
||||
@ -160,6 +165,13 @@ The files must now be provided first when calling `tokenizer.train(files, traine
|
||||
- Actually add special tokens in tokenizers implementations ([acef252](https://github.com/huggingface/tokenizers/commit/acef252dacc43adc414175cfc325668ad1488753))
|
||||
|
||||
|
||||
[#956]: https://github.com/huggingface/tokenizers/pull/956
|
||||
[#1008]: https://github.com/huggingface/tokenizers/pull/1008
|
||||
[#1009]: https://github.com/huggingface/tokenizers/pull/1009
|
||||
[#1047]: https://github.com/huggingface/tokenizers/pull/1047
|
||||
[#1055]: https://github.com/huggingface/tokenizers/pull/1055
|
||||
[#1051]: https://github.com/huggingface/tokenizers/pull/1051
|
||||
[#1052]: https://github.com/huggingface/tokenizers/pull/1052
|
||||
[#938]: https://github.com/huggingface/tokenizers/pull/938
|
||||
[#939]: https://github.com/huggingface/tokenizers/pull/939
|
||||
[#952]: https://github.com/huggingface/tokenizers/pull/952
|
||||
|
@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "tokenizers",
|
||||
"version": "0.12.1",
|
||||
"version": "0.13.0",
|
||||
"description": "",
|
||||
"main": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
|
Reference in New Issue
Block a user