Neural Abstract Reasoner: claims that they has achieved 80% accuracy on this dataset about arc HOT 5 CLOSED

"As this work is still in progress, these are preliminary results evaluated on grids up to 10×10."

I guess this 80% is only on the grids of small sizes, which are disproportionately the "easy" tasks involving mere rotation/mirroring.

from arc.

Has anyone tried implementing this paper? I can't find a working demonstration anywhere.

from arc.

from arc.

I think there are several strange things about their paper.

• In a later presentation (2021) they say "NAR achieves 61.13% accuracy on the Abstraction and Reasoning Corpus" with no further explanation. Is that measure on the entire dataset (i.e. all grid sizes)? Otherwise, why is number different? They use the exact same graphs as motivation in their poster as in the old article, yet with different numbers as the result, see: https://eucys2021.usal.es/computing-03-2021/
• As far as I understand it they evaluate on the public test set, yet they compare it to the Kaggle competition which ran on completely different, hidden, tasks.
• They claim to solve 78,8% of 100 hidden tasks but don't explain how they get .8 when the test are binary.
• There is no discussion on what the impact is when excluding all larger grids, this is especially relevant when they are comparing agains the Kaggle competition.
• There is no source code, no one (AFAIK) has reproduced the results, and there is no official benchmark against the hidden test set.

It's possible they have devised an approach that is better than the previous state-of-the-art, but at this point I find it hard to take their numbers at face value.

from arc.

from arc.