mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fixing hanging error while acquiring GIL from custom pretokenizer during training. (#470)

Browse Source 
* Fixing hanging error while acquiring GIL from custom pretokenizer
during training.

Fixes #469

* cleanup

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
This commit is contained in:
Nicolas Patry
2020-10-20 20:23:39 +02:00
committed by GitHub
parent 91f602f744
commit 180371d929
2 changed files with 32 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
bindings/python/src/tokenizer.rs
View File

@ -751,7 +751,9 @@ impl PyTokenizer {
}
fn train(&mut self, trainer: &PyTrainer, files: Vec<String>) -> PyResult<()> {
ToPyResult(self.tokenizer.train_and_replace(trainer, files)).into()
let gil = Python::acquire_gil();
gil.python()
.allow_threads(|| ToPyResult(self.tokenizer.train_and_replace(trainer, files)).into())
}
#[args(pair = "None", add_special_tokens = true)]