mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-03 19:28:20 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,258 Commits 1 Branch 0 Tags
2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a
Commit Graph

499 Commits

Author SHA1 Message Date
Nicolas Patry
2ccd16bf5c Adding a new tests for PreTokenizer.custom. 
This example is more illustrative of what's doable for custom
PreTokenizer.
 2020-10-15 10:07:48 -04:00
Anthony MOI
f94a274702 Python - Update CHANGELOG and bump version for release 2020-10-13 14:45:21 -04:00
Nicolas Patry
88556790e7 Fixing a bug where long tokenizer files would be incorrectly deserialized (#459) 
* Fixing a bug where long tokenizer files would be incorrectly
deserialized

- Add a bunch of tests to check deserialization behaviour
- One tests also confirms current Single deserialization of Sequence.

* Better test locations for Windows + no file dependency in Python binding
Rust side.

* Adressing @n1t0 comments.
 2020-10-13 18:44:24 +02:00
Anthony MOI
3bb794681c Python - Use 1.46.0 for now 2020-10-09 13:40:35 -04:00
Anthony MOI
83e11a8de4 Python - Update dependencies for release 2020-10-09 13:09:35 -04:00
Anthony MOI
4f4ba4a11a Python - Bump version for 0.9.0 release 2020-10-09 13:00:19 -04:00
Nicolas Patry
fbca797b3d Fixing Trainer with u8 instead of chars. (#452) 
* Fixing Trainer with u8 instead of chars.

Now check both optimized and unoptimized encodings schemes for Unigram.

* Small fixes.

* Fixing makefile.
 2020-10-09 18:57:14 +02:00
Nicolas Patry
dd9fda5d05 Bump rc version. 2020-10-06 11:04:36 +02:00
Anthony MOI
aebf510c5a Python - Update CHANGELOG and bump to 0.9.0.rc1 2020-09-29 10:24:24 -04:00
Anthony MOI
ff57504972 Python - Add some more test for TemplateProcessing 2020-09-29 10:09:10 -04:00
Nicolas Patry
6c25bb729b Update __init__.pyi 2020-09-29 10:09:10 -04:00
Anthony MOI
1070eb471e Python - Update bindings for TemplateProcessing 2020-09-29 10:09:10 -04:00
Dagmawi Moges
7f8b357b92 Fixed Dead Link: Build your own #435 (#436) 
* Fixed Dead Link: Build your own #435

* Update bindings/python/README.md

Co-authored-by: Anthony MOI <xn1t0x@gmail.com>
 2020-09-25 09:41:31 -04:00
Anthony MOI
a0a163fd62 Remove unwanted file 2020-09-24 14:05:47 -04:00
Anthony MOI
171a042ee0 Python - Bump version for dev4 release 2020-09-24 10:16:18 -04:00
Nicolas Patry
a410903051 Upgrading to black 20.8b1 2020-09-24 09:27:30 -04:00
Anthony MOI
8308508577 Python - Update bindings for Replace Normalizer 2020-09-24 08:05:57 -04:00
Nicolas Patry
598ce61229 Removed now wrong code in convert.py, fixed strange black magic. 2020-09-24 08:57:02 +02:00
Nicolas Patry
95cc8c47ad Changed rust api for merges, that is now Vec<(String, String)> 2020-09-24 08:57:02 +02:00
Nicolas Patry
36832bfa12 from_files -> from_file everywhere 
- read_files -> read_file
- from_file pure rust impl in python bindings
- Fix some typing in python binding
- Added {BPE,WordLevel,WordPiece}.from_file tests.
 2020-09-24 08:57:02 +02:00
Nicolas Patry
9672995a56 We use 19.10b0 not 20 here... 2020-09-24 08:57:02 +02:00
Nicolas Patry
35ee1968c0 Black *Version* check. 2020-09-24 08:57:02 +02:00
Nicolas Patry
9b1ef9d895 Black pre-commit after rebase. 2020-09-24 08:57:02 +02:00
Nicolas Patry
acd4a7599f Black. 2020-09-24 08:57:02 +02:00
Nicolas Patry
8f8156fd2c Adressing first pass of comments. 2020-09-24 08:57:02 +02:00
Nicolas Patry
1cd4824273 Black on pyi file. 2020-09-24 08:57:02 +02:00
Nicolas Patry
60c1e25910 New version. Staticmethods need to return a IntoPy<PyObject> 
which is non trivial for PyClassInitializer. Instead I added a lower
staticmethod that returns raw objects, and the `from_file(s)` methods
are implemented directly in Python.
 2020-09-24 08:57:02 +02:00
Nicolas Patry
98a30eead1 Temp work to make the APIs uniform (build from memory by default). 2020-09-24 08:57:02 +02:00
Anthony MOI
b24a2fc178 Some suggestions from @narsil 2020-09-23 15:50:01 -04:00
Anthony MOI
31b81f109b Python - Fix for PySlice on Windows 2020-09-23 15:50:01 -04:00
Anthony MOI
b9a051f464 Python - Update some missing typings 2020-09-23 15:50:01 -04:00
Anthony MOI
7492a1d698 Python - Update typings for NormalizedString 2020-09-23 15:50:01 -04:00
Anthony MOI
0b448f46d4 Python - Update typings for PreTokenizedString 2020-09-23 15:50:01 -04:00
Anthony MOI
b1097a988f Python - Improved example with custom components 2020-09-23 15:50:01 -04:00
Anthony MOI
0a930ef1d8 Python - Update bindings for PreTokenizer 2020-09-23 15:50:01 -04:00
Anthony MOI
53aad4eca0 Python - Update support for custom Decoder 2020-09-23 15:50:01 -04:00
Anthony MOI
08a3128515 Python - Add bindings for some Model methods 2020-09-23 15:50:01 -04:00
Anthony MOI
5276238b1b Python - Add bindings for PostProcessor.process 2020-09-23 15:50:01 -04:00
Anthony MOI
b6e7a6e2f7 Python - Update PyNormalizer interface 2020-09-23 15:50:01 -04:00
Anthony MOI
bd8f25ee2c Python - Add support for custom PreTokenizer 2020-09-23 15:50:01 -04:00
Anthony MOI
8d04b22278 Python - Add support for custom Normalizer 2020-09-23 15:50:01 -04:00
Anthony MOI
9245471dcd Python - Add bindings for PreTokenizedString 2020-09-23 15:50:01 -04:00
Anthony MOI
003d2ac6fb Python - Update PyToken bindings 2020-09-23 15:50:01 -04:00
Anthony MOI
fce6998dcf Python - Add bindings for NormalizedString 2020-09-23 15:50:01 -04:00
Anthony MOI
e4b10e0fd9 Python - Add RefMutGuard to safely share &mut 2020-09-23 15:50:01 -04:00
Anthony MOI
a42e13a644 Setup black format in pyproject.toml 2020-09-23 11:58:35 -04:00
Nicolas Patry
9d3a93db5b Going back for not fuse_unk by default for BPE, but add a flag to 
enable it.
 2020-09-22 16:27:09 -04:00
Anthony MOI
940f8bd8fa Update PyO3 (#426) 2020-09-22 12:00:20 -04:00
Nicolas Patry
c536b4992b Move to dev3 build. 2020-09-22 08:21:38 +02:00
Nicolas Patry
07197e8e35 Move to spm_precompiled 0.1.2 for smaller binary string. 2020-09-22 08:21:38 +02:00