mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
555d44c47a6ebca921a6bd6192ff95674f15c6a5
tokenizers/bindings/python/py_src/tokenizers
History
Nicolas Patry cc5fb01a2f Decode stream python (#1678) 
* Python binding for decode stream

Different API because Python cannot handle lifetimes properly.

* Clippy.
2024-11-15 12:06:22 +01:00
..
decoders
Decode stream python (#1678)
2024-11-15 12:06:22 +01:00
implementations
Fixing doc. (#1499)
2024-04-17 09:32:40 +02:00
models
Fix 'dictionnary' typo (#1511)
2024-06-11 15:43:47 +02:00
normalizers
Add bytelevel normalizer to fix decode when adding tokens to BPE (#1555)
2024-07-15 12:12:03 +02:00
pre_tokenizers
[MINOR:TYPO] Fix docstrings (#1653)
2024-11-05 16:25:06 +01:00
processors
[remove black] And use ruff (#1436)
2024-03-12 11:24:21 +01:00
tools
[remove black] And use ruff (#1436)
2024-03-12 11:24:21 +01:00
trainers
[remove black] And use ruff (#1436)
2024-03-12 11:24:21 +01:00
__init__.py
make tests happy
2023-09-05 15:37:09 +00:00
__init__.pyi
fix pylist (#1673)
2024-11-05 16:24:23 +01:00