This website requires JavaScript.
Explore
Help
Sign In
mii
/
tokenizers
Watch
1
Star
0
Fork
0
You've already forked tokenizers
mirror of
https://github.com/mii443/tokenizers.git
synced
2025-08-23 00:35:35 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
daf3fcc976c5e021ceef7c3d2a83af632a5634f7
tokenizers
/
bindings
/
python
/
examples
History
Quentin Lhoest
e76f900bc0
Faster
datasets
train example
...
Using .iter() is much faster than accessing using row ids
2023-03-23 11:24:30 +01:00
..
custom_components.py
pyo3 v0.18 migration (
#1173
)
2023-03-08 11:27:47 +01:00
example.py
Updating python formatting. (
#1079
)
2022-10-05 15:29:33 +02:00
train_bert_wordpiece.py
Updating python formatting. (
#1079
)
2022-10-05 15:29:33 +02:00
train_bytelevel_bpe.py
Updating python formatting. (
#1079
)
2022-10-05 15:29:33 +02:00
train_with_datasets.py
Faster
datasets
train example
2023-03-23 11:24:30 +01:00
using_the_visualizer.ipynb
Add a visualization utility to render tokens and annotations in a notebook (
#508
)
2020-12-04 10:25:56 -05:00