mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • c46ec97855 Update README Anthony MOI 2019-12-03 17:26:20 -05:00
  • 75232c0f06 Fix setup.py Anthony MOI 2019-12-03 16:20:20 -05:00
  • 499f5507df Bump versions for 0.0.3 release Anthony MOI 2019-12-03 16:11:45 -05:00
  • ec2ed483a3 Improve python readme with training example Anthony MOI 2019-12-03 16:11:03 -05:00
  • eaafb22511 Add bindings for Trainer in Python Anthony MOI 2019-12-03 15:54:15 -05:00
  • 310a2af76b Add BPE empty constructor Anthony MOI 2019-12-03 15:39:54 -05:00
  • 0324beea57 BpeTrainer is a Trainer Anthony MOI 2019-12-03 15:39:33 -05:00
  • 466555bade Add Trainer trait and Tokenizer.train Anthony MOI 2019-12-03 15:38:45 -05:00
  • 768eb9b920 bpe::Error implements std::error::Error Anthony MOI 2019-12-03 15:23:08 -05:00
  • 5011523e99 Update python readme Anthony MOI 2019-12-03 10:25:55 -05:00
  • 5f31ac3f75 Python release CI (#2) MOI Anthony 2019-12-02 19:04:25 -05:00
  • 1a52cda912 Fix yaml indent Morgan Funtowicz 2019-11-26 11:23:12 +01:00
  • f9ccf62301 Try updating to official rust Github Action to avoid missing rust components. Morgan Funtowicz 2019-11-26 11:20:12 +01:00
  • 78e7591780 Fix Cargo.toml not found in Rust workflow Morgan Funtowicz 2019-11-26 10:59:34 +01:00
  • 5db08ac15d Update wheel building Anthony MOI 2019-11-29 22:36:17 -05:00
  • 27ac65c466 Remove onig dependency Anthony MOI 2019-11-29 21:35:16 -05:00
  • d1b6b14bd7 Attempt fix workflows Anthony MOI 2019-11-29 19:28:49 -05:00
  • 989e9b03ca Ignore some python files Anthony MOI 2019-11-27 12:22:01 -05:00
  • 428890d6e0 Basic python setuptools Anthony MOI 2019-11-27 12:21:37 -05:00
  • e49abab747 Python - Add Decoder/PreTokenizer standalone capabilities Anthony MOI 2019-11-26 17:52:19 -05:00
  • d565bbf309 Container - Add ability to execute Anthony MOI 2019-11-26 17:51:26 -05:00
  • 5c6834f363 Added GitHub Action workflow for Rust Funtowicz Morgan 2019-11-26 09:47:48 +00:00
  • f4369b312d Python - Add ability to create custom Decoder Anthony MOI 2019-11-25 19:14:07 -05:00
  • d7ba6802df Update gitignore Anthony MOI 2019-11-25 15:35:54 -05:00
  • 512e85dfda Update python README Anthony MOI 2019-11-24 00:55:13 -05:00
  • bafdc5e157 Code style Anthony MOI 2019-11-24 00:52:48 -05:00
  • 6437c40235 Python - PoC Custom PreTokenizer Anthony MOI 2019-11-24 00:52:13 -05:00
  • b081e6ca04 Python - Also expose default classes Anthony MOI 2019-11-24 00:35:05 -05:00
  • bd1aa80d8a Python - Custom PreTokenizer backbone Anthony MOI 2019-11-23 23:59:33 -05:00
  • 891fc12de2 Python - Update example with new format Anthony MOI 2019-11-22 21:09:17 -05:00
  • 8fbe3c2662 Python - Add decoders Anthony MOI 2019-11-22 21:08:57 -05:00
  • e44f52024c Python - Set a PreTokenizer in a model Anthony MOI 2019-11-22 21:01:52 -05:00
  • 9b71c8f8de Python - BPE construction Anthony MOI 2019-11-22 20:57:54 -05:00
  • f6a9b57b5b Python - Add pre_tokenizers module Anthony MOI 2019-11-22 20:56:50 -05:00
  • 39a6d04c53 Improve Python bindings Anthony MOI 2019-11-22 17:57:36 -05:00
  • 663644e041 Fix ByteLevel Decoder Anthony MOI 2019-11-21 16:50:25 -05:00
  • 634415c098 Add a parallel capable cache for BPE Anthony MOI 2019-11-21 16:09:07 -05:00
  • 070fd08583 Update python example Anthony MOI 2019-11-21 11:57:57 -05:00
  • c28a83cdc4 Update python bindings Anthony MOI 2019-11-21 11:55:07 -05:00
  • 6853e6c904 Tokenizer decoding Anthony MOI 2019-11-21 11:54:54 -05:00
  • 2419c14e42 ByteLevel is also a Decoder Anthony MOI 2019-11-21 11:52:55 -05:00
  • 56e37475c3 Add Decoder to Tokenizer Anthony MOI 2019-11-21 11:51:43 -05:00
  • 3ec26b332c Add Tokenizer token_to_id/id_to_token Anthony MOI 2019-11-20 17:28:28 -05:00
  • 8b3d7d1aa0 Add vocab/merge arguments to example.py Anthony MOI 2019-11-20 16:47:02 -05:00
  • 98323d1f21 Update readme and fix example Anthony MOI 2019-11-19 19:38:57 -05:00
  • 351d526e1e Basic python bindings Anthony MOI 2019-11-19 19:31:33 -05:00
  • 39afc64e13 impl PreTokenizer for Whitespace Anthony MOI 2019-11-19 18:14:57 -05:00
  • 2d7c5f04f8 Fix readme indentation MOI Anthony 2019-11-18 16:34:13 -05:00
  • 1b32560067 Update readme with simple example Anthony MOI 2019-11-18 16:31:35 -05:00
  • 872aa86b71 Basic cli for testing Anthony MOI 2019-11-18 15:47:35 -05:00
  • 4e5106989f Ability to load a BPE model from files Anthony MOI 2019-11-18 10:00:53 -05:00
  • 0b450d62ff Add ByteLevel pre tokenizer Anthony MOI 2019-11-17 00:40:22 -05:00
  • a55dccafb5 Add BPE training Anthony MOI 2019-11-17 00:28:36 -05:00
  • 1c7dcebca7 Add BPE tokenization Anthony MOI 2019-11-17 00:27:30 -05:00
  • b2ba864248 Move whitespace pre tokenizer Anthony MOI 2019-11-16 22:42:02 -05:00
  • 1294f400dc Add folder structure Anthony MOI 2019-11-16 22:40:51 -05:00
  • 7b8b765269 Add Tokenizer interface Anthony MOI 2019-11-16 22:36:44 -05:00
  • 195423fe11 Rust install Anthony MOI 2019-11-01 19:45:00 -04:00
  • 9f15d2c165 Node readme Anthony MOI 2019-11-01 19:44:44 -04:00
  • 05cbb32eca Python readme Anthony MOI 2019-11-01 19:42:36 -04:00
  • 6d91bf4005 Update node bindings Anthony MOI 2019-11-01 19:23:22 -04:00
  • fd7ec39367 Update python bindings Anthony MOI 2019-11-01 18:56:55 -04:00
  • 9fd10ca1c5 Simple whitespace tokenizer Anthony MOI 2019-11-01 18:31:05 -04:00
  • 57a1ce7e1d Node bindings backbone Anthony MOI 2019-11-01 16:39:03 -04:00
  • 8448d50e6f Quick improvement over python bindings Anthony MOI 2019-11-01 16:08:10 -04:00
  • 5d37cfde7f Python bindings backbone Anthony MOI 2019-11-01 15:02:19 -04:00
  • 5f57ee9f0e Global gitignore Anthony MOI 2019-11-01 14:55:08 -04:00
  • 7dbee7157f add gitignore Anthony MOI 2019-11-01 13:56:08 -04:00
  • 2b72a5737f Basic bin+lib setup Anthony MOI 2019-11-01 13:54:17 -04:00
  • b9b519c84a Initial commit Anthony MOI 2019-11-01 13:52:44 -04:00