mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-07 13:18:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update CHANGELOGs

Browse Source
This commit is contained in:
Anthony MOI
2020-03-05 11:09:16 -05:00
parent b2e5f54b6f
commit d8e7a830b2
2 changed files with 11 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
bindings/python/CHANGELOG.md
View File

@@ -3,6 +3,12 @@
## Changes: ## Changes:
- Keep only one progress bar while reading files during training. This is better for use-cases with - Keep only one progress bar while reading files during training. This is better for use-cases with
a high number of files as it avoids having too many progress bar on screen. a high number of files as it avoids having too many progress bar on screen.
- `add_prefix_space` option of the `ByteLevel` `PreTokenizer` has been moved to a `Normalizer`
## How to migrate:
- Use the `ByteLevel` `Normalizer` with `add_prefix_space=True` in addition to the `PreTokenizer`.
The `PreTokenizer` does not handle this option anymore. This fixes some issues with the offsets
being wrong if this option was on.
# v0.6.0 # v0.6.0

5
tokenizers/CHANGELOG.md
View File

@@ -4,6 +4,11 @@
- Keep only one progress bar while reading files during training. This is better for use-cases with - Keep only one progress bar while reading files during training. This is better for use-cases with
a high number of files as it avoids having too many progress bar on screen. a high number of files as it avoids having too many progress bar on screen.
- Improve BPE and WordPiece builders. - Improve BPE and WordPiece builders.
- `ByteLevel` is also a `Normalizer` and handles the `add_prefix_space` option at this level now.
This fixes some issues with the offsets being wrong if this option was on.
## How to migrate:
- Use the `ByteLevel` as a `Normalizer` if `add_prefix_space` is required.
# v0.8.0 # v0.8.0