15 Commits

Author	SHA1	Message	Date
Nicolas Patry	02ad59edc1	Never gonna run around and desert you	2023-05-12 16:27:06 +02:00
Roy Hvaara	4d520c9664	Ignore Cargo.lock for subfolders (#1131)	2022-12-25 11:35:47 +01:00
Tal Perry	8916b6bb27	Add a visualization utility to render tokens and annotations in a notebook (#508) ... * Draft functionality of visualization * Added comments to make code more intelligble * polish the styles * Ensure colors are stable and comment the css * Code clean up * Made visualizer importable and added some docs * Fix styling * implement comments from PR * Fixed the regex for UNK tokens and examples in notebook * Converted docs to google format * Added a notebook showing multiple languages and tokenizers * Added visual indication of chars that are tokenized with >1 token * Reorganize things a bit and fix import * Update docs Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>	2020-12-04 10:25:56 -05:00
Nicolas Patry	81bb4f6da3	Actually adding docs.	2020-11-02 17:07:27 -05:00
Pierric Cistac	88354b0e40	gitignore vscode workspace files	2020-06-15 18:08:41 -04:00
Anthony MOI	36e3c28a23	Ignore .vim folder	2020-05-01 17:11:54 -04:00
Anthony MOI	023566fbbb	Python - Add some tests utils	2020-04-01 17:25:55 -04:00
Anthony MOI	a80d0a302f	Update .gitignores for data folder	2020-03-16 10:36:41 -04:00
thomwolf	6d0e3ba8f1	fix imports	2020-01-07 18:54:21 -05:00
Anthony MOI	70a17ce330	Ignore some more python files	2020-01-06 15:23:21 -05:00
Anthony MOI	5b8cd00d21	Ignore Python package files	2020-01-06 13:04:45 -05:00
Evan Pete Walsh	9f1421a04b	remove Cargo.lock (#7)	2019-12-23 21:22:42 -08:00
Anthony MOI	989e9b03ca	Ignore some python files	2019-11-27 12:22:01 -05:00
Anthony MOI	d7ba6802df	Update gitignore	2019-11-25 15:35:54 -05:00
Anthony MOI	5f57ee9f0e	Global gitignore	2019-11-01 14:55:08 -04:00