mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python - Update CHANGELOG and bump version for 0.9.4

Browse Source
This commit is contained in:
Anthony MOI
2020-11-09 16:13:21 -05:00
committed by Anthony MOI
parent d3d9f2c76b
commit 75b41dab0f
5 changed files with 19 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
bindings/python/CHANGELOG.md
View File

@ -4,6 +4,16 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.9.4]
### Fixed
- [#492]: Fix `from_file` on `BertWordPieceTokenizer`
- [#498]: Fix the link to download `sentencepiece_model_pb2.py`
- [#500]: Fix a typo in the docs quicktour
### Changed
- [#506]: Improve Encoding mappings for pairs of sequence
## [0.9.3] ## [0.9.3]
### Fixed ### Fixed
@ -259,6 +269,11 @@ delimiter (Works like `.split(delimiter)`)
- Fix a bug with the IDs associated with added tokens. - Fix a bug with the IDs associated with added tokens.
- Fix a bug that was causing crashes in Python 3.5 - Fix a bug that was causing crashes in Python 3.5
[#506]: https://github.com/huggingface/tokenizers/pull/506
[#500]: https://github.com/huggingface/tokenizers/pull/500
[#498]: https://github.com/huggingface/tokenizers/pull/498
[#492]: https://github.com/huggingface/tokenizers/pull/492
[#481]: https://github.com/huggingface/tokenizers/pull/481 [#481]: https://github.com/huggingface/tokenizers/pull/481
[#480]: https://github.com/huggingface/tokenizers/pull/480 [#480]: https://github.com/huggingface/tokenizers/pull/480
[#477]: https://github.com/huggingface/tokenizers/pull/477 [#477]: https://github.com/huggingface/tokenizers/pull/477

2
bindings/python/Cargo.lock generated
View File

@ -893,7 +893,7 @@ dependencies = [
[[package]] [[package]]
name = "tokenizers-python" name = "tokenizers-python"
version = "0.9.3" version = "0.9.4"
dependencies = [ dependencies = [
"env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", "env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.79 (registry+https://github.com/rust-lang/crates.io-index)", "libc 0.2.79 (registry+https://github.com/rust-lang/crates.io-index)",

2
bindings/python/Cargo.toml
View File

@ -1,6 +1,6 @@
[package] [package]
name = "tokenizers-python" name = "tokenizers-python"
version = "0.9.3" version = "0.9.4"
authors = ["Anthony MOI <m.anthony.moi@gmail.com>"] authors = ["Anthony MOI <m.anthony.moi@gmail.com>"]
edition = "2018" edition = "2018"

2
bindings/python/py_src/tokenizers/__init__.py
View File

@ -1,4 +1,4 @@
__version__ = "0.9.3" __version__ = "0.9.4"
from typing import Tuple, Union, Tuple, List from typing import Tuple, Union, Tuple, List
from enum import Enum from enum import Enum

2
bindings/python/setup.py
View File

@ -6,7 +6,7 @@ extras["testing"] = ["pytest"]
setup( setup(
name="tokenizers", name="tokenizers",
version="0.9.3", version="0.9.4",
description="Fast and Customizable Tokenizers", description="Fast and Customizable Tokenizers",
long_description=open("README.md", "r", encoding="utf-8").read(), long_description=open("README.md", "r", encoding="utf-8").read(),
long_description_content_type="text/markdown", long_description_content_type="text/markdown",