mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Modify Processor trait to support chaining. (#1054)

Browse Source 
0 modifications yet, everything will consume the vector.
Every test should be green without any modifications.
This commit is contained in:
Nicolas Patry
2022-08-24 19:49:23 +02:00
committed by GitHub
parent b1c9bc68b5
commit 460bdded80
8 changed files with 149 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
bindings/node/native/src/processors.rs
View File

@ -22,16 +22,15 @@ impl tk::PostProcessor for Processor {
.added_tokens(is_pair)
}
fn process(
fn process_encodings(
&self,
encoding: Encoding,
pair_encoding: Option<Encoding>,
encodings: Vec<Encoding>,
add_special_tokens: bool,
) -> tk::Result<Encoding> {
) -> tk::Result<Vec<Encoding>> {
self.processor
.as_ref()
.ok_or("Uninitialized PostProcessor")?
.process(encoding, pair_encoding, add_special_tokens)
.process_encodings(encodings, add_special_tokens)
}
}