mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python - Expose encode method on Model

Browse Source
This commit is contained in:
Anthony MOI
2020-03-18 23:04:32 -04:00
parent 8de6ef5a37
commit a397a1da63
6 changed files with 139 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
bindings/python/src/encoding.rs
View File

@ -4,7 +4,7 @@ use crate::error::PyError;
use pyo3::prelude::*;
use pyo3::types::*;
use pyo3::{PyObjectProtocol, PySequenceProtocol};
use tk::tokenizer::PaddingDirection;
use tk::tokenizer::{Offsets, PaddingDirection};
#[pyclass(dict)]
#[repr(transparent)]