300 Commits

Author	SHA1	Message	Date
Anthony MOI	1c4593cad4	Python - Remove warning on unused Token	2019-12-13 15:28:48 -05:00
Anthony MOI	e93cc62a71	Python - Handle kwargs for bert modules	2019-12-13 15:28:29 -05:00
Anthony MOI	3355be89cd	Python - Update examples and improve errors	2019-12-13 14:37:29 -05:00
Anthony MOI	7cf4b3a6cd	Python - Rewrite PyDecoder and PyPreTokenizer	2019-12-13 12:20:25 -05:00
Anthony MOI	2a0ad97809	Python - Update API to allow failure	2019-12-13 12:20:05 -05:00
Anthony MOI	1c7be358b7	Python - Better error conversions	2019-12-13 12:14:27 -05:00
Anthony MOI	7cb2fe2ea0	Bump version	2019-12-10 18:01:07 -05:00
Anthony MOI	b4b31d73cd	Expose vocabulary size	2019-12-10 16:20:31 -05:00
Anthony MOI	6c294c60b0	Python - Add Encoding repr + improve example	2019-12-10 15:18:07 -05:00
Anthony MOI	99773d9ce4	Python - Add encoding getters	2019-12-10 15:17:41 -05:00
Anthony MOI	8cedc5f1f6	Update Python bindings for Encoding	2019-12-10 12:38:36 -05:00
Anthony MOI	018f57f054	Python - Update example	2019-12-09 12:51:05 -05:00
Anthony MOI	849272d44f	Python - add missing modules exports	2019-12-09 12:50:53 -05:00
Anthony MOI	3979096c52	Python - add BasicPreTokenizer	2019-12-09 12:50:09 -05:00
Anthony MOI	d60d24a378	Python - Add WordPiece model	2019-12-09 12:49:44 -05:00
Anthony MOI	5eba30835d	Python - Add WordPiece decoder	2019-12-09 12:49:17 -05:00
Anthony MOI	c46ec97855	Update README	2019-12-03 17:26:20 -05:00
Anthony MOI	75232c0f06	Fix setup.py	2019-12-03 16:20:20 -05:00
Anthony MOI	499f5507df	Bump versions for 0.0.3 release	2019-12-03 16:11:45 -05:00
Anthony MOI	ec2ed483a3	Improve python readme with training example	2019-12-03 16:11:03 -05:00
Anthony MOI	eaafb22511	Add bindings for Trainer in Python	2019-12-03 15:54:15 -05:00
Anthony MOI	5011523e99	Update python readme	2019-12-03 10:26:19 -05:00
MOI Anthony	5f31ac3f75	Python release CI (#2)	2019-12-02 19:04:25 -05:00
Anthony MOI	5db08ac15d	Update wheel building	2019-11-29 22:36:17 -05:00
Anthony MOI	27ac65c466	Remove onig dependency	2019-11-29 21:35:16 -05:00
Anthony MOI	428890d6e0	Basic python setuptools	2019-11-27 12:21:37 -05:00
Anthony MOI	e49abab747	Python - Add Decoder/PreTokenizer standalone capabilities	2019-11-26 17:52:19 -05:00
Anthony MOI	d565bbf309	Container - Add ability to execute	2019-11-26 17:51:26 -05:00
Anthony MOI	f4369b312d	Python - Add ability to create custom Decoder	2019-11-25 19:14:07 -05:00
Anthony MOI	512e85dfda	Update python README	2019-11-24 00:55:13 -05:00
Anthony MOI	6437c40235	Python - PoC Custom PreTokenizer	2019-11-24 00:52:13 -05:00
Anthony MOI	b081e6ca04	Python - Also expose default classes	2019-11-24 00:35:05 -05:00
Anthony MOI	bd1aa80d8a	Python - Custom PreTokenizer backbone	2019-11-23 23:59:33 -05:00
Anthony MOI	891fc12de2	Python - Update example with new format	2019-11-22 21:09:17 -05:00
Anthony MOI	8fbe3c2662	Python - Add decoders	2019-11-22 21:08:57 -05:00
Anthony MOI	e44f52024c	Python - Set a PreTokenizer in a model	2019-11-22 21:01:52 -05:00
Anthony MOI	9b71c8f8de	Python - BPE construction	2019-11-22 20:57:54 -05:00
Anthony MOI	f6a9b57b5b	Python - Add pre_tokenizers module	2019-11-22 20:56:50 -05:00
Anthony MOI	39a6d04c53	Improve Python bindings ... This is an attempt at actually exposing the same structure that we use in the Rust lib. This will allow Python to instantiate Model/PreTokenizer/... with their own arguments, combining everything without relying on parsed kwargs.	2019-11-22 17:57:36 -05:00
Anthony MOI	663644e041	Fix ByteLevel Decoder ... The join was done after replacing bytes and building subwords, which was preventing bytes across these subwords to be merged correctly. We need to join first.	2019-11-21 16:50:25 -05:00
Anthony MOI	070fd08583	Update python example	2019-11-21 11:57:57 -05:00
Anthony MOI	c28a83cdc4	Update python bindings	2019-11-21 11:55:07 -05:00
Anthony MOI	3ec26b332c	Add Tokenizer token_to_id/id_to_token	2019-11-20 17:28:28 -05:00
Anthony MOI	8b3d7d1aa0	Add vocab/merge arguments to example.py	2019-11-20 16:47:02 -05:00
Anthony MOI	98323d1f21	Update readme and fix example	2019-11-19 19:38:57 -05:00
Anthony MOI	351d526e1e	Basic python bindings	2019-11-19 19:31:37 -05:00
Anthony MOI	05cbb32eca	Python readme	2019-11-01 19:42:36 -04:00
Anthony MOI	fd7ec39367	Update python bindings	2019-11-01 18:56:55 -04:00
Anthony MOI	8448d50e6f	Quick improvement over python bindings	2019-11-01 16:08:10 -04:00
Anthony MOI	5d37cfde7f	Python bindings backbone	2019-11-01 15:02:19 -04:00