mii/tokenizers
1
0
Fork 0
mirror of https://github.com/mii443/tokenizers.git synced 2025-08-22 16:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix unigram.rs test_sample() (#1244)

Browse Source 
87230bb59b/tokenizers/tests/unigram.rs (LL71C1-L71C53)

When running cargo test --release, the above line causes an error.

referring to 87230bb59b/tokenizers/src/models/unigram/lattice.rs (L138)

It seems that lattice::from should only take 3 arguments.
If i had to guess, it should be Lattice::from("ABC", 0, 2);
This change makes cargo test --release pass without error.
This commit is contained in:
Chris Ha
2023-05-11 00:04:34 +09:00
committed by GitHub
parent 87230bb59b
commit 4518b0f7f2
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tokenizers/tests/unigram.rs
View File

@ -68,7 +68,7 @@ fn test_train_unigram_from_file() {
#[cfg(not(debug_assertions))]
#[test]
fn test_sample() {
let mut lattice = Lattice::from("ABC", 0, 1, 2);
let mut lattice = Lattice::from("ABC", 0, 2);
lattice.insert(0, 1, 1.0, 3); // A
lattice.insert(1, 1, 1.2, 4); // B
lattice.insert(2, 1, 1.5, 5); // C