mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-23 00:35:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

pyo3: update to 0.20 (#1386)

Browse Source 
Co-authored-by: Mike Lui <mikelui@meta.com>
This commit is contained in:
Michael Lui
2024-01-11 06:03:13 -10:00
committed by GitHub
parent 8939d4e26d
commit 888dd4bc65
4 changed files with 6 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
bindings/python/src/tokenizer.rs
View File

@ -7,7 +7,6 @@ use pyo3::exceptions;
use pyo3::intern;
use pyo3::prelude::*;
use pyo3::types::*;
use pyo3::AsPyPointer;
use tk::models::bpe::BPE;
use tk::tokenizer::{
Model, PaddingDirection, PaddingParams, PaddingStrategy, PostProcessor, TokenizerImpl,