keyvank / femtogpt Goto Github PK

Add infrastructures for CPU training. Let's use OpenCL!

Hi,
Assert fails in tensor/mod.rs when number of heads = number of layers. Not sure if it's not a good number but in nanogpt, equal values are supported - see the cpu section (apples to oranges so please ignore if it's not a good comparison):

https://github.com/karpathy/nanoGPT

It would be good if people could store their entire model+descritption+training_data in a single file so that others could easily infer or fine tune them.

Let's make femtoGPT a library!

Is it normal for the loss to be jittering?

I've been training my model for a few hours (like 6?), at the start the loss was mostly only decreasing but at around 3.00 it started jittering and the jittering only gets more intense. Currently my loss ranges from ~2.67 to ~2.92.

I'm training on my own dataset (10k lines, ~150KB) with 78 unique characters and 312k parameters (not sure if that matters)

Hello, I'm trying to run femtoGPT on my rx 6600 under Ubuntu Linux, I've installed the required Rocm OpenCL drivers but, when I run the program using

cargo run --release --features gpu

I get an index out of bounds exception

thread 'main' panicked at 'index out of bounds: the len is 0 but the index is 0', src/graph/gpu/mod.rs:119:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Edit

This is the log with RUST_BACKTRACE=1

0: rust_begin_unwind
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:593:5
1: core::panicking::panic_fmt
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:67:14
2: core::panicking::panic_bounds_check
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panicking.rs:162:5
3: femto_gpt::graph::gpu::GpuGraph::new
4: femto_gpt::main
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

hi @keyvank ,
let's say I want to add a new decoder layer (the one that gets constructed as part of 0..num_layers loop) at run time after the gpt::new() call, how do I go about it ? As I understand you are just pushing the computations one by one incremented by tensorid so adding a layer at a later point of time will need incrementing the ids for the next layers as well (for example adding one more decoder layer along with all the sub layers like attention etc means incrementing the vocab out and other variables outside the for loop ?)

Also why keep computations as a btree when in reality it's being used more like a Vec as we are not even using the id against which each computation is stored (please correct me if I missed something :) )

thread 'main' panicked at src/tokenizer/simple.rs:43:47:
called Option::unwrap() on a None value
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Hi,
It's mentioned that this has 300k parameters. How to increase the number of parameters. Is it by increasing the number of layers/heads ? (also how are the total parameters calculated)

Thank you for attempting a pure Rust implementation of GPT. Helps a lot to understand things without having to understand python voodoo.

Edit: Also if possible having some comments around important parts of the code will help a lot from learning perspective. Would be great if you wrote a blog post on it :)

Apologies if my question is stupid, is it at all possible to create a model so we can run this generated model on , say, ollama ?

Currently, I have been doing .unwrap()s all over the code which is very bad. It's good to have different error types and raise them instead of panicking.

E.g: TensorError when a faulty tensor operation happens, or GraphError on faulty computation graphs or GptError for core LLM related errors.

Right now, the code is very undocumented. As femtoGPT is also an educational library, it's good to have comprehensive guides, documentations and commentsm

Hi @keyvank ,
Instead of letting the model generate the whole text in infer, how do we make it complete a user provided text?

I'm guessing it should be by filling context vector in infer method.

Find the best num-layers/embedding-degree/learning-rate/etc. in terms of accuracy and training time.

Could you guys provide a description of how to use this tokenizer ? I tried by myself but i couldn't figure out on how to make it work.

Thanks :)

We will need to have more clever methods of tokenization in femtoGPT. Possibly, it's good to have a SentencePiece model reader.

This will reduce our number of params and make our model faster.

Trade-off could be complexity in femtoGPT.

I can assign you @cutoken to this if you are interested.

Maybe create a tutorial for training a GPT model on Rust and put it on codeproject.com? Or somewhere similar?