From 0e890d0d05ba49a7ce1ce25baf2bfb7d23fd6a37 Mon Sep 17 00:00:00 2001 From: Anthony MOI Date: Thu, 21 May 2020 16:33:30 -0400 Subject: [PATCH] Update CHANGELOGs --- bindings/python/CHANGELOG.md | 3 +++ tokenizers/CHANGELOG.md | 1 + 2 files changed, 4 insertions(+) diff --git a/bindings/python/CHANGELOG.md b/bindings/python/CHANGELOG.md index af1795a3..3803ba99 100644 --- a/bindings/python/CHANGELOG.md +++ b/bindings/python/CHANGELOG.md @@ -13,6 +13,7 @@ This adds some methods to easily save/load an entire tokenizer (`from_str`, `fro ### Added - [#272]: Serialization of the `Tokenizer` and all the parts (`PreTokenizer`, `Normalizer`, ...). This adds some methods to easily save/load an entire tokenizer (`from_str`, `from_file`). +- [#273]: `Tokenizer` and its parts are now pickable ### Changed - Improved errors generated during truncation: When the provided max length is too low are @@ -178,6 +179,8 @@ delimiter (Works like `.split(delimiter)`) - Fix a bug with the IDs associated with added tokens. - Fix a bug that was causing crashes in Python 3.5 +[#273]: https://github.com/huggingface/tokenizers/pull/273 +[#272]: https://github.com/huggingface/tokenizers/pull/272 [#249]: https://github.com/huggingface/tokenizers/pull/249 [#239]: https://github.com/huggingface/tokenizers/pull/239 [#236]: https://github.com/huggingface/tokenizers/pull/236 diff --git a/tokenizers/CHANGELOG.md b/tokenizers/CHANGELOG.md index 49eb8e10..f8b4359c 100644 --- a/tokenizers/CHANGELOG.md +++ b/tokenizers/CHANGELOG.md @@ -105,6 +105,7 @@ advised, but that's not the question) split up in multiple bytes - [#174]: The `LongestFirst` truncation strategy had a bug +[#272]: https://github.com/huggingface/tokenizers/pull/272 [#249]: https://github.com/huggingface/tokenizers/pull/249 [b770f36]: https://github.com/huggingface/tokenizers/commit/b770f364280af33efeffea8f0003102cda8cf1b7 [#236]: https://github.com/huggingface/tokenizers/pull/236