30 Commits

Author	SHA1	Message	Date
Anthony MOI	eaafb22511	Add bindings for Trainer in Python	2019-12-03 15:54:15 -05:00
Anthony MOI	5011523e99	Update python readme	2019-12-03 10:26:19 -05:00
MOI Anthony	5f31ac3f75	Python release CI (#2)	2019-12-02 19:04:25 -05:00
Anthony MOI	5db08ac15d	Update wheel building	2019-11-29 22:36:17 -05:00
Anthony MOI	27ac65c466	Remove onig dependency	2019-11-29 21:35:16 -05:00
Anthony MOI	428890d6e0	Basic python setuptools	2019-11-27 12:21:37 -05:00
Anthony MOI	e49abab747	Python - Add Decoder/PreTokenizer standalone capabilities	2019-11-26 17:52:19 -05:00
Anthony MOI	d565bbf309	Container - Add ability to execute	2019-11-26 17:51:26 -05:00
Anthony MOI	f4369b312d	Python - Add ability to create custom Decoder	2019-11-25 19:14:07 -05:00
Anthony MOI	512e85dfda	Update python README	2019-11-24 00:55:13 -05:00
Anthony MOI	6437c40235	Python - PoC Custom PreTokenizer	2019-11-24 00:52:13 -05:00
Anthony MOI	b081e6ca04	Python - Also expose default classes	2019-11-24 00:35:05 -05:00
Anthony MOI	bd1aa80d8a	Python - Custom PreTokenizer backbone	2019-11-23 23:59:33 -05:00
Anthony MOI	891fc12de2	Python - Update example with new format	2019-11-22 21:09:17 -05:00
Anthony MOI	8fbe3c2662	Python - Add decoders	2019-11-22 21:08:57 -05:00
Anthony MOI	e44f52024c	Python - Set a PreTokenizer in a model	2019-11-22 21:01:52 -05:00
Anthony MOI	9b71c8f8de	Python - BPE construction	2019-11-22 20:57:54 -05:00
Anthony MOI	f6a9b57b5b	Python - Add pre_tokenizers module	2019-11-22 20:56:50 -05:00
Anthony MOI	39a6d04c53	Improve Python bindings ... This is an attempt at actually exposing the same structure that we use in the Rust lib. This will allow Python to instantiate Model/PreTokenizer/... with their own arguments, combining everything without relying on parsed kwargs.	2019-11-22 17:57:36 -05:00
Anthony MOI	663644e041	Fix ByteLevel Decoder ... The join was done after replacing bytes and building subwords, which was preventing bytes across these subwords to be merged correctly. We need to join first.	2019-11-21 16:50:25 -05:00
Anthony MOI	070fd08583	Update python example	2019-11-21 11:57:57 -05:00
Anthony MOI	c28a83cdc4	Update python bindings	2019-11-21 11:55:07 -05:00
Anthony MOI	3ec26b332c	Add Tokenizer token_to_id/id_to_token	2019-11-20 17:28:28 -05:00
Anthony MOI	8b3d7d1aa0	Add vocab/merge arguments to example.py	2019-11-20 16:47:02 -05:00
Anthony MOI	98323d1f21	Update readme and fix example	2019-11-19 19:38:57 -05:00
Anthony MOI	351d526e1e	Basic python bindings	2019-11-19 19:31:37 -05:00
Anthony MOI	05cbb32eca	Python readme	2019-11-01 19:42:36 -04:00
Anthony MOI	fd7ec39367	Update python bindings	2019-11-01 18:56:55 -04:00
Anthony MOI	8448d50e6f	Quick improvement over python bindings	2019-11-01 16:08:10 -04:00
Anthony MOI	5d37cfde7f	Python bindings backbone	2019-11-01 15:02:19 -04:00