mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-12-06 04:38:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Roberta PostProcessor (#111)

Browse Source 
* Added RobertaProcessor on Rust side.

Required to match the double separator token in the middle of pairs.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* Fix typo in RobertaProcessing method declaration

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* Correctly include RobertProcessor in the Python binding

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* Roberta doesnt use token_type_ids so let's set everything to 0

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* Attempt to make it works on Node side too.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

* fix js bindings / `npm run lint`

* Make RustFmt happy.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
This commit is contained in:
Funtowicz Morgan
2020-02-03 10:39:48 +00:00
committed by GitHub
parent 8138ece1a6
commit 6524f09e99
9 changed files with 175 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
bindings/python/src/lib.rs
View File

@@ -59,6 +59,7 @@ fn decoders(_py: Python, m: &PyModule) -> PyResult<()> {
fn processors(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_class::<processors::PostProcessor>()?;
m.add_class::<processors::BertProcessing>()?;
m.add_class::<processors::RobertaProcessing>()?;
Ok(())
}