mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-04 11:48:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix BPE continuing_subword_prefix

Browse Source
This commit is contained in:
Anthony MOI
2021-03-16 16:50:42 -04:00
committed by Anthony MOI
parent f5e9bb89b7
commit 0fe9214f44
2 changed files with 49 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/CHANGELOG.md
View File

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- [#652]: Fix offsets for `Precompiled` corner case
- [#656]: Fix BPE `continuing_subword_prefix`
## [0.10.1]
@@ -307,6 +308,7 @@ delimiter (Works like `.split(delimiter)`)
- Fix a bug that was causing crashes in Python 3.5
[#656]: https://github.com/huggingface/tokenizers/pull/656
[#652]: https://github.com/huggingface/tokenizers/pull/652
[#621]: https://github.com/huggingface/tokenizers/pull/621
[#620]: https://github.com/huggingface/tokenizers/pull/620