Logo
Explore Help
Sign In
mii/tokenizers
1
0
Fork 0
You've already forked tokenizers
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-08 05:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
275ee6d4c44b405db01eddf0077b2ac95f83696b
tokenizers/bindings/python/py_src/tokenizers
History
Nicolas Patry 792d618006 Adding a new "Replace" normalizer that takes a string and replaces it
with another String (for now).
2020-09-18 12:24:39 +02:00
..
decoders
Move Python source to subdirectory.
2020-07-25 23:40:47 +02:00
implementations
Improvements on spm parity: (#401)
2020-09-15 22:21:02 +02:00
models
Address @n1t0 comments.
2020-09-02 16:32:50 +02:00
normalizers
Adding a new "Replace" normalizer that takes a string and replaces it
2020-09-18 12:24:39 +02:00
pre_tokenizers
Adding a new pre_tokenizer: Digits.
2020-09-03 21:03:45 +02:00
processors
TemplateProcessing - Add @narsil suggestions
2020-09-10 15:04:19 -04:00
trainers
Expose the trainer to Python bindings.
2020-09-02 13:38:14 -04:00
__init__.py
Python - Bump version for 0.9.0.dev1 release
2020-09-02 22:31:01 -04:00
__init__.pyi
Remove Tokenizer::normalize
2020-08-19 12:42:12 -04:00
Powered by Gitea Version: 1.24.5 Page: 127ms Template: 9ms
English
Bahasa Indonesia Deutsch English Español Français Gaeilge Italiano Latviešu Magyar nyelv Nederlands Polski Português de Portugal Português do Brasil Suomi Svenska Türkçe Čeština Ελληνικά Български Русский Українська فارسی മലയാളം 日本語 简体中文 繁體中文(台灣) 繁體中文(香港) 한국어
Licenses API