mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

[MINOR:TYPO] Fix docstrings (#1653)

Browse Source 
* [MINOR:TYPO] Update pre_tokenizers.rs

* [MINOR:TYPO] Update __init__.pyi
This commit is contained in:
Christopher Akiki
2024-11-05 16:25:06 +01:00
committed by GitHub
parent 5e223ceb48
commit 57884ebaa2
2 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi
View File

@ -421,7 +421,7 @@ class Split(PreTokenizer):
Args: Args:
pattern (:obj:`str` or :class:`~tokenizers.Regex`): pattern (:obj:`str` or :class:`~tokenizers.Regex`):
A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`. A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`, If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
otherwise we consider is as a string pattern. For example `pattern="|"` otherwise we consider is as a string pattern. For example `pattern="|"`
means you want to split on `|` (imagine a csv file for example), while means you want to split on `|` (imagine a csv file for example), while

2
bindings/python/src/pre_tokenizers.rs
View File

@ -334,7 +334,7 @@ impl PyWhitespaceSplit {
/// ///
/// Args: /// Args:
/// pattern (:obj:`str` or :class:`~tokenizers.Regex`): /// pattern (:obj:`str` or :class:`~tokenizers.Regex`):
/// A pattern used to split the string. Usually a string or a a regex built with `tokenizers.Regex`. /// A pattern used to split the string. Usually a string or a regex built with `tokenizers.Regex`.
/// If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`, /// If you want to use a regex pattern, it has to be wrapped around a `tokenizer.Regex`,
/// otherwise we consider is as a string pattern. For example `pattern="|"` /// otherwise we consider is as a string pattern. For example `pattern="|"`
/// means you want to split on `|` (imagine a csv file for example), while /// means you want to split on `|` (imagine a csv file for example), while