Tal Perry 8916b6bb27 Add a visualization utility to render tokens and annotations in a notebook (#508) ...

* Draft functionality of visualization

* Added comments to make code more intelligble

* polish the styles

* Ensure colors are stable and comment the css

* Code clean up

* Made visualizer importable and added some docs

* Fix styling

* implement comments from PR

* Fixed the regex for UNK tokens and examples in notebook

* Converted docs to google format

* Added a notebook showing multiple languages and tokenizers

* Added visual indication of chars that are tokenized with >1 token

* Reorganize things a bit and fix import

* Update docs

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>

2020-12-04 10:25:56 -05:00

..

node.inc

Doc - Reorganize API Reference

2020-11-02 17:07:27 -05:00

python.inc

Add a visualization utility to render tokens and annotations in a notebook (#508)

2020-12-04 10:25:56 -05:00

reference.rst

Doc - Reorganize API Reference

2020-11-02 17:07:27 -05:00

rust.inc

Doc - Improve quicktour support for multi-lang

2020-11-02 17:07:27 -05:00