mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 16:49:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Clippy fixes. (#846)

Browse Source 
* Clippy fixes.

* Drop support for Python 3.6

* Remove other 3.6

* Re-enabling caches for build (5h + seems too long and issue seems
solved)

https://github.com/actions/virtual-environments/issues/572

* `npm audit fix`.

* Fix yaml ?

* Pyarrow issue fixed: https://github.com/huggingface/datasets/pull/2268

* Installing dev libraries.

* Install python dev elsewhere ?

* Typo.

* No sudo.

* ...

* Testing the GH again.

* Maybe v2 will fix ?

* Fixing tests on MacOS Python 3.8+
This commit is contained in:
Nicolas Patry
2021-12-15 15:55:48 +01:00
committed by GitHub
parent 1dc19e0dd4
commit c1100ec542
23 changed files with 365 additions and 402 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
bindings/python/tests/utils.py
View File

@ -98,6 +98,10 @@ def doc_pipeline_bert_tokenizer(data_dir):
)
# On MacOS Python 3.8+ the default was modified to `spawn`, we need `fork` in tests.
mp.set_start_method("fork")
def multiprocessing_with_parallelism(tokenizer, enabled: bool):
"""
This helper can be used to test that disabling parallelism avoids dead locks when the