mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-05 04:08:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,204 Commits 1 Branch 0 Tags
003d2ac6fb8552be3b023a9c0ea51dd927429eda
Commit Graph

7 Commits

Author SHA1 Message Date
Nicolas Patry
816632c9fa Removing --release compat test. 
- Leaving the one that checks that sampling follows the expected
distribution.
- Marking the python Unigram.train(..) test as slow
- The python Unigram.train(..) test now uses `big.txt` file.
 2020-09-02 13:38:14 -04:00
Nicolas Patry
d0366529b7 Use a smaller train file. 2020-09-02 13:38:14 -04:00
Nicolas Patry
7b5c2b92c6 Fixing test dependency. 2020-09-02 13:38:14 -04:00
Anthony MOI
aa3b39f692 Python - Tests for parallelism with multiprocessing 
Co-authored-by: Evan Pete Walsh <epwalsh10@gmail.com>
 2020-06-23 11:25:39 -04:00
Anthony MOI
7fd7dfd113 Python - Test CharBPETokenizer 2020-04-01 17:25:56 -04:00
Anthony MOI
dbc23e20a9 Python - Test Models 2020-04-01 17:25:55 -04:00
Anthony MOI
023566fbbb Python - Add some tests utils 2020-04-01 17:25:55 -04:00