* First pass on automatic stubbing our python files.
* And now modifying all rust docs to be visible in Pyi files.
* Better assert fail message.
* Fixing github workflow.
* Removing types not exported anymore.
* Fixing `Tokenizer` signature.
* Disabling auto __init__.py.
* Re-enabling some types.
* Don't overwrite non automated __init__.py
* Automated most __init__.py
* Restubbing after rebase.
* Fixing env for tests.
* Install blakc in the env.
* Use PY35 target in stub.py
Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
* New PR to fix#270 (not #157).
Reduce drastically the number of required compilation flags.
I think it's good enough for merge right now. We disable progress
altogether when the `progressbar` flag is disabled which is perfectly
fine compared to not being able to build.
Future PR could include.
- Better encapsulation of `progress` in training call sites (less direct
calls to `indicatif` and common code for `setup_progress`, `finalize`
and so on.
- We can have a raw `print` Progress bar when compilation flag is
disabled ?
- Having better control of progressbars in bindings would require use to
change a bunch of code around which might be overkill in the short term.
Either we start by defining a trait for our ProgressBar, and the
bindings can implement the traits with custom `tqdm` and `cli-progress`
(It's not even 100% sure it's doable)
- The easiest way would be to enable some sort of iterator in Rust
so that calling of progressbars can happen in client code which would
be the most lenient for all plateforms. The hard part is that
leveraging parallelism in that setting would be hard probably.
* Remove external visibility of progressbar.
* Remove dead import.
* Moving to manylinux2010 and remove nightly on Windows.
* Add build for manylinux2014 for powerpc and aarch64 + Python v3.9
* Also add support for IBM mainframe
* try with env variables
* Move extra builds to their own workflow
Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
* Fixing hanging error while acquiring GIL from custom pretokenizer
during training.
Fixes#469
* cleanup
Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
* Fixing a bug where long tokenizer files would be incorrectly
deserialized
- Add a bunch of tests to check deserialization behaviour
- One tests also confirms current Single deserialization of Sequence.
* Better test locations for Windows + no file dependency in Python binding
Rust side.
* Adressing @n1t0 comments.
