mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-05 12:18:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Re-enabling Custom Serialize

Browse Source
This commit is contained in:
Nicolas Patry
2020-08-21 17:39:00 +02:00
committed by Anthony MOI
parent 6887c0f04d
commit 1994dcad6e
1 changed files with 1 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
bindings/python/src/pre_tokenizers.rs
View File

@@ -382,9 +382,7 @@ impl Serialize for PyPreTokenizerWrapper {
ser.end() ser.end()
} }
PyPreTokenizerWrapper::Wrapped(inner) => inner.serialize(serializer), PyPreTokenizerWrapper::Wrapped(inner) => inner.serialize(serializer),
PyPreTokenizerWrapper::Custom(_) => { PyPreTokenizerWrapper::Custom(inner) => inner.serialize(serializer),
unreachable!("Custom pretokenizers are currently disabled, how did you get here?")
}
} }
} }
} }