mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-17 01:28:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update CHANGELOGs

Browse Source
This commit is contained in:
Anthony MOI
2020-03-02 14:37:17 -05:00
parent f10f77d2e1
commit 4deeb9511f
2 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
bindings/python/CHANGELOG.md
View File

@@ -1,6 +1,9 @@
# v0.6.0 (not published yet)
Fixes:
## Changes:
- Big improvements in speed for BPE (Both training and tokenization) ([#165](https://github.com/huggingface/tokenizers/pull/165))
## Fixes:
- Some default tokens were missing from `BertWordPieceTokenizer` (cf [#160](https://github.com/huggingface/tokenizers/issues/160))
- There was a bug in ByteLevel PreTokenizer that caused offsets to be wrong if a char got split up
in multiple bytes. (cf [#156](https://github.com/huggingface/tokenizers/pull/156))