Commit Graph

4 Commits

Author SHA1 Message Date
8916b6bb27 Add a visualization utility to render tokens and annotations in a notebook (#508)
* Draft functionality of visualization

* Added comments to make code more intelligble

* polish the styles

* Ensure colors are stable and comment the css

* Code clean up

* Made visualizer importable and added some docs

* Fix styling

* implement comments from PR

* Fixed the regex for UNK tokens and examples in notebook

* Converted docs to google format

* Added a notebook showing multiple languages and tokenizers

* Added visual indication of chars that are tokenized with >1 token

* Reorganize things a bit and fix import

* Update docs

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
2020-12-04 10:25:56 -05:00
58e1d8de67 Python - Improve documentation for trainers 2020-11-23 11:52:51 -05:00
d2fc0e4836 Doc - Update API Reference for Encoding 2020-11-02 17:07:27 -05:00
5cb954db11 Doc - Reorganize API Reference 2020-11-02 17:07:27 -05:00