mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix a bug when adding special tokens

Browse Source 
If we add special tokens that are part of the vocabulary of the model, the tokens aren't added to the tokenizer, which then built an empty regex. This completely break the tokenization
This commit is contained in:
Anthony MOI
2019-12-26 14:32:50 -05:00
parent d93d4fc3cd
commit d1e59e09bf
1 changed files with 6 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
tokenizers/src/tokenizer/mod.rs
View File

@ -577,7 +577,12 @@ impl Tokenizer {
})
.collect::<Vec<_>>();
self.split_re = Some(regex::Regex::new(&format!(r"({})", added_tokens.join("|"))).unwrap());
if added_tokens.is_empty() {
self.split_re = None;
} else {
self.split_re =
Some(regex::Regex::new(&format!(r"({})", added_tokens.join("|"))).unwrap());
}
// Return the number of added tokens
tokens.len() - ignored