How is the simulated video length related to the object states? about phyre HOT 3 CLOSED

I went through some tasks in the within protocol, fold 0 test set data. I found some task-action pairs which are labeled as SOLVED very weird, it seems that the simulated videos end before the objects enter stable states, leading to wrong labels.

Examples:

• task 230, act 20: mp4
• task 236, act 393: mp4

My conjecture is that, the simulator sees the cyan and the blue object in contact lasting for 3s, so it assumes the simulation should end and assign a SOLVED label? The good news is, it seems that most of such errors (if they indeed are) only happen in this one task family.

Another possibility is that I misunderstood the goal condition. Should the goal state be a stable state, or if the two target objects just touch each other once and then separate is still considered a success?

from phyre.

Hi there!
The condition for task being marked as solved is whether two target objects are in touching relation for 3 seconds. It does allow for some weird solutions where the objects are not stable but still are in contact for 3 seconds. That is pretty rare though

from phyre.

I agree, it's indeed very rare. I train a model only looking at the last frame to determine whether the task is solved, and can achieve 99% accuracy.
Thanks for the answer!

from phyre.