6 Commits

Author	SHA1	Message	Date
Nicolas Patry	36832bfa12	from_files -> from_file everywhere ... - read_files -> read_file - from_file pure rust impl in python bindings - Fix some typing in python binding - Added {BPE,WordLevel,WordPiece}.from_file tests.	2020-09-24 08:57:02 +02:00
Nicolas Patry	9b1ef9d895	Black pre-commit after rebase.	2020-09-24 08:57:02 +02:00
Nicolas Patry	8f8156fd2c	Adressing first pass of comments.	2020-09-24 08:57:02 +02:00
Nicolas Patry	98a30eead1	Temp work to make the APIs uniform (build from memory by default).	2020-09-24 08:57:02 +02:00
Anthony MOI	aa3b39f692	Python - Tests for parallelism with multiprocessing ... Co-authored-by: Evan Pete Walsh <epwalsh10@gmail.com>	2020-06-23 11:25:39 -04:00
Anthony MOI	7fd7dfd113	Python - Test CharBPETokenizer	2020-04-01 17:25:56 -04:00