mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add a visualization utility to render tokens and annotations in a notebook (#508)

Browse Source 
* Draft functionality of visualization

* Added comments to make code more intelligble

* polish the styles

* Ensure colors are stable and comment the css

* Code clean up

* Made visualizer importable and added some docs

* Fix styling

* implement comments from PR

* Fixed the regex for UNK tokens and examples in notebook

* Converted docs to google format

* Added a notebook showing multiple languages and tokenizers

* Added visual indication of chars that are tokenized with >1 token

* Reorganize things a bit and fix import

* Update docs

Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
This commit is contained in:
Tal Perry
2020-12-04 16:25:56 +01:00
committed by GitHub
parent 5549fc4837
commit 8916b6bb27
8 changed files with 1651 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -4,7 +4,7 @@
.vim
.env
target
.idea
Cargo.lock
/data
@ -17,6 +17,7 @@ __pycache__
pip-wheel-metadata
*.egg-info
*.so
/bindings/python/examples/.ipynb_checkpoints
/bindings/python/build
/bindings/python/dist