diff --git a/README.md b/README.md index 94476acb..66f7dd77 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,17 @@ -# Tokenizers +
+
+
+
+
+
+ Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. @@ -14,6 +27,12 @@ versatility. original sentence that corresponds to a given token. - Does all the pre-processing: Truncate, Pad, add the special tokens your model needs. +
+
+
+
+
+ ## Bindings We provide bindings to the following languages (more to come!):