mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Word indices are None for special tokens

Browse Source
This commit is contained in:
Anthony MOI
2020-04-09 09:52:02 -04:00
parent e7949fcb84
commit 3ad1360210
10 changed files with 66 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/src/encoding.rs
View File

@ -62,7 +62,7 @@ impl Encoding {
}
#[getter]
fn get_words(&self) -> Vec<u32> {
fn get_words(&self) -> Vec<Option<u32>> {
self.encoding.get_words().to_vec()
}