mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-06 20:58:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,059 Commits 1 Branch 0 Tags
10a39ba6b48bc942078f04a64710f89cdce1e360
Commit Graph

406 Commits

Author SHA1 Message Date
Sebastian Pütz
10a39ba6b4 Add in-place train. 2020-08-04 15:59:33 -04:00
Sebastian Pütz
ac8af63f70 Trainers don't need Arc. 2020-08-04 15:59:33 -04:00
Anthony MOI
363adedb4c Fixes and cleanup, suggestions by @n1t0. 2020-08-04 15:59:33 -04:00
Sebastian Pütz
f6adcf0e7c Remove typetag, bump deps. 2020-08-04 15:59:33 -04:00
Sebastian Puetz
16f75d9efc Ensure serialization works in all expected ways. 2020-08-04 15:59:33 -04:00
Sebastian Puetz
aaf8e932b1 Remove Send + Sync requirements from Model. 2020-08-04 15:59:33 -04:00
Sebastian Puetz
42b810488f Hide generics 2020-08-04 15:59:33 -04:00
Sebastian Pütz
d62adf7195 Remove Container, changes to PyDecoder, cloneable Tokenizer. 
* derive Clone on Tokenizer and AddedVocabulary.
* Replace Container with Arc wrapper for Decoders.
* Prefix Rust Decoder types with Py.
* Rename PyDecoder to CustomDecoder.
* Change panic in serializing custom decoder to exception.
* Re-enable training with cloneable Tokenizer.
* Remove unsound Container, use Arc wrappers instead.
 2020-08-04 15:59:33 -04:00
Sebastian Pütz
11e86a16c5 Remove Container from PostProcessors, replace with Arc. 
* prefix the Python types in Rust with Py.
* remove unsound Container wrappers, replace with Arc.
 2020-08-04 15:59:33 -04:00
Sebastian Pütz
b411443128 Remove Container from PreTokenizers, replace with Arc. 
* prefix the Python types in Rust with Py, rename PyPretokenizer
  to CustomPretokenizer
* remove unsound Container wrappers, replace with Arc
* change panic on trying to (de-)serialize custom pretokenizer to
  exception
 2020-08-04 15:59:33 -04:00
Sebastian Pütz
08b8c48127 Remove Container from Normalizers, replace with Arc. 
* prefix the Python types in Rust with Py
* remove unsound Container wrappers, replace with Arc
 2020-08-04 15:59:33 -04:00
Sebastian Pütz
83a52c8080 Replace Model and Trainer Containers. 
* Implement changes necessary from generic Model in Tokenizer.
* Temporarily disable training in Python since Clone can't be
  derived for Model until all components have been replaced.
* Prefix Python types in Rust with Py.
 2020-08-04 15:59:33 -04:00
Anthony MOI
dad70e8e85 Implement suggestions by @sebpuetz 
Co-authored-by: Sebastian Pütz <sebastian.puetz@uni-tuebingen.de>
 2020-08-03 16:18:59 -04:00
Anthony MOI
7833965dc4 Update Python bindings with new interface 2020-08-03 16:18:59 -04:00
Anthony MOI
904ff24382 New API for PreTokenizer and Model + refactor Tokenizer - WIP 2020-08-03 16:18:59 -04:00
Sebastian Pütz
27e326ab2b Fix deadlocks with custom python components. 2020-08-03 16:17:17 -04:00
Sebastian Pütz
0d7c232f95 Move Python source to subdirectory. 
This allows testing versions not built in-place. Otherwise
importing (or testing) in the package root fails without develop
builds.
Replace maturin with setuptools_rust since maturin fails with
proper project structure.
 2020-07-25 23:40:47 +02:00
Anthony MOI
c901f86d52 Python - Bump version for 0.8.1 2020-07-20 16:33:48 -04:00
Anthony MOI
157feed9a5 Python - Bump version for 0.8.1.rc2 2020-07-17 13:12:23 -04:00
Setu Shah
1f2cc6ee73 Include license in PyPI package 2020-07-16 14:20:32 -04:00
Anthony MOI
5be375eaea Update CHANGELOGs and bump version for python release 2020-07-06 15:21:47 -04:00
Anthony MOI
e874641cf9 Merge pull request #333 from huggingface/fix-added-tokens 
Python - Fix Added token deserialization
 2020-07-06 14:52:37 -04:00
Anthony MOI
2194970679 Merge pull request #330 from huggingface/bert-normalization 
Improve BertNormalizer behavior
 2020-07-06 14:52:23 -04:00
Anthony MOI
d33af1a3be Python - Fix Added token deserialization 2020-07-06 14:46:12 -04:00
Anthony MOI
7a95ffc4fa BertNormalizer has same behavior than original implem 2020-07-06 13:55:18 -04:00
Anthony MOI
8bf482cecc Improve parallelism tracking and warning 2020-07-06 13:05:14 -04:00
आलोक
6fe284dd8d Use supplied UNK token even when vocab absent 
If a vocab file isn't provided the supplied unk token (different from [UNK]) gets ignored and later throws an error:
Exception: WordPiece error: Missing [UNK] token from the vocabulary
when trying to encode an input string with an unknown token.
 2020-07-05 19:01:04 +05:30
Anthony MOI
6349ca51b3 Python - Bump version for 0.8.0 release 2020-06-26 16:12:26 -04:00
Anthony MOI
8ae1982149 Finally it will be rc4 for transformers 2020-06-26 15:36:08 -04:00
Anthony MOI
5a653869af Try local version for transformers 2020-06-26 15:19:00 -04:00
Anthony MOI
1a08b21329 Python - Bump version for 0.8.0.transformers release 2020-06-26 14:37:22 -04:00
Anthony MOI
bb668bc439 Try with target_family = unix 2020-06-23 16:52:21 -04:00
Anthony MOI
f8b1630aa6 Update CHANGELOGs 2020-06-23 13:32:21 -04:00
Anthony MOI
aa3b39f692 Python - Tests for parallelism with multiprocessing 
Co-authored-by: Evan Pete Walsh <epwalsh10@gmail.com>
 2020-06-23 11:25:39 -04:00
Anthony MOI
ae743f5dc1 Python - Automatically disable parallelism after fork 2020-06-22 20:31:52 -04:00
Anthony MOI
5d20322319 Rust - Fix optional parallelism with par_bridge 2020-06-22 20:31:52 -04:00
Anthony MOI
dce52621c6 Rust - Make parallelism optional 2020-06-22 20:31:52 -04:00
Anthony MOI
74d812d401 Python - Bump version to 0.8.0.rc3 for release 2020-06-22 12:54:31 -04:00
Anthony MOI
c02d4e2202 Python - Improve AddedToken interface 2020-06-19 17:53:46 -04:00
Anthony MOI
a14cd7b219 Python - Bump version to 0.8.0.rc2 for release 2020-06-19 10:48:53 -04:00
Anthony MOI
898a4a812e Python - Make AddedToken pickable 2020-06-19 10:34:11 -04:00
Anthony MOI
63edb95130 Python - Update AddedToken repr 2020-06-19 10:18:55 -04:00
Anthony MOI
4c7a0ff4ec Update CHANGELOGs 2020-06-18 14:50:16 -04:00
Anthony MOI
fc63d56eab AddedVocabulary - Add tests, update bindings + various tweaks 2020-06-18 14:50:16 -04:00
Anthony MOI
c6f633eb1c Rust - Fix/Tweak AddedVocabulary + Fix python tests 2020-06-16 14:42:53 -04:00
Anthony MOI
397cc539da Rust - Add AddedVocabulary + normalized option on AddedToken 2020-06-16 14:42:53 -04:00
Anthony MOI
fb964adfdb Python - Bump version to 0.8.0.rc1 for release 2020-06-11 14:24:34 -04:00
Anthony MOI
847651445e Fix build-wheels.sh script for manylinux wheels 
Before this change, we added the `.so` files from previous version in
the `.whl` files of later versions.

Fix #301
 2020-06-11 12:43:40 -04:00
Anthony MOI
433a311887 Update CHANGELOGs 2020-06-09 17:33:41 -04:00
Anthony MOI
794759b56d Python - Improve truncation/padding management 2020-06-09 17:33:41 -04:00