lalalune / arcprize Goto Github PK

Instead of trying to predict all of the tokens at once, we should predict some, keep the ones with high confident, and try again with a modify attention mask.

The data needs to get harder incrementally. It's pointless to train a fresh model on something a 100 IQ person probably couldn't solve. Starting with ultra simple tasks until loss is down, and then introducing more and more complex tasks into the mix. Eventually taking out the tasks which are under the threshold difficulty of the real problem.

Training is slow. We should do whatever we can to improve memory and training speed.

We should evaluate every 100 epochs to get a sense of improvement

Right now we are using a LOT of padding tokens. Will this make things bad? I don't know, it's pretty sparse. We could try implementing sparse attention.

sinusoidal position encoding has some flaws that rope could solve

Wave Function Collapse has a knowable pattern. Discovering this patten from the output image should teach our model the kind of program search we really want.

Right now we are training with no dropout, which is guaranteed to cause some trouble

We are kind of blindly just matrix transforming, color swapping and augmenting.

• Implement visualization of current augmentation to verify that it makes sense
• Add noise patterns underneath black as an additional augmentation time
• Add some augmentation that replaces token 0 with one of the colors not used in the puzzle

Review our teacher forcing strategy.

One idea that might be interesting is to set it to 1 - loss. So we force until out loss below 0, then start to back off. By the end the model shouldn't care about sequence order.

For 16x16 and under, our max grid cell count is 256, so we could make the context much shorter in this case.

src = src.to(device)
output = model(src)
target = src[:, model.num_context_tokens:].reshape(-1)
loss = criterion(output.view(-1, num_tokens + 1), target.view(-1))

Your model looks at the entire source?