mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Preparing rc1 release. (#1056)

Browse Source 
* Preparing rc1 release.

* Fixing test_alignment_methods

* Fixing the overflowing sequence_id issue (LayoutLMv2 tests caught this).

* Adding overly complex overflowing test.
This commit is contained in:
Nicolas Patry
2022-09-12 16:07:06 +02:00
committed by GitHub
parent 06025e4ca1
commit 7bfab48979
15 changed files with 417 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/setup.py
View File

@ -8,7 +8,7 @@ extras["dev"] = extras["testing"]
setup(
name="tokenizers",
version="0.12.1.dev0",
version="0.13.0.dev0",
description="Fast and Customizable Tokenizers",
long_description=open("README.md", "r", encoding="utf-8").read(),
long_description_content_type="text/markdown",