![](https://private-user-images.githubusercontent.com/159022025/331331890-375f2c3b-a5ad-4330-b1ce-7ca2f623003e.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQyNzg4NjIsIm5iZiI6MTcyNDI3ODU2MiwicGF0aCI6Ii8xNTkwMjIwMjUvMzMxMzMxODkwLTM3NWYyYzNiLWE1YWQtNDMzMC1iMWNlLTdjYTJmNjIzMDAzZS5qcGc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODIxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgyMVQyMjE2MDJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iZjFmMTIyODRiYTE2MzA5YmU3MTM2YjBiNmVmOGYxMWRhM2IzMTEwY2JlM2EzMTk4Yzc0MDNjZWI1YmU4MDZlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.co3S7f-imR9ggcXfe3ho1waVBDnsAPjoDCxS3lsJnaE)
Simple tool for training n-gram language model. Inspired by this course.
use n_gram::*;
fn main() {
// Initializing model
let config = Config::default();
let mut model = Model::new(config);
// Loading and tokenizing corpus
let corpus = tiny_corpus()
.iter()
.map(|t| sos(eos(tokenize(t.to_owned()))))
.collect::<Vec<_>>();
model.train(corpus);
// Now you are ready to generate something
let mut tokens = sos(tokenize("The quick".to_owned()));
let max = 10; // max number of generated tokens
model.generate(&mut tokens, max);
// Save model
model.save("model.json").unwrap();
// Reset model
model.reset();
// Load model back
model.load("model.json").unwrap();
}
I've trained a trigram model on 20000 samples from the Tiny Stories dataset. Here are some examples of generated text:
- "__sos__ Once upon a time a mom, a dad, a big sister, and a little girl below shouted, "Look Mama! A talking cloud!" The little girl opened her hand, and the monkey happily ate it all in one day. She was so kind he said yes and showed him the pin. "I poked you with this. It is a storm. The waves were so tall and wide, it seemed like something was calling her to come to an end eventually. They all had an incredible songbird inside. Billy was happy and excited. __eos__"
- "__sos__ Once upon a time there was a light girl with a basket. She then sent the basket to the washing machine. While the laundry was all hung up, Daisy and her family were getting ready to fly it, it suddenly flew away! The lion felt bad for being rude. He said, "It's my p leasure. It's important to remember to forgive. __eos__"
- "__sos__ Once upon a time a family lived in a stream with many stones on the ground, it glistened in the sunshine. From that day forth they were always with her and learn with her and waved goodbye to Mommy. The bus driver was happy and flew away happily. Timmy felt proud of their pictures. __eos__"
cargo add n_gram
If you want to save & load your models:
cargo add n_gram --features=saveload
If you want to load tiny corpus for training:
cargo add n_gram --features=corpus