Updated image model weights for CARLA 0.9.10 about 2020_carla_challenge HOT 6 CLOSED

hm this seems a bit odd - the weights that are in the README score a 8.94 on the carla challenge (0.9.10.1) leaderboard link

some things i've noticed about the model is that it does pretty poorly on towns 1-2, but can get a nonzero score on the other towns

from 2020_carla_challenge.

Yeah I've been trying to wrap my head around this too - there's a pretty big disconnect between the 0.9.10.1 Leaderboard scores on the website and the performance I'm seeing on my end. As far as I know, the repository seems very plug-and-play: download the repository, install CARLA 0.9.10.1, download the weights and run the image agent. Regardless, I'm still seeing 0% on nearly every route given in leaderboard/data/routes_* no matter which town is being used.

I was wondering if it might be due to something like package version mismatch or floating point precision, but my pytorch/pytorch-lightning/torch/torchvision versions line up with what's specified in carla_project/requirements.txt.

from 2020_carla_challenge.

have you visualized the model's predictions via the HAS_DISPLAY env variable? I'll also try to investigate this over the weekend!

from 2020_carla_challenge.

Thanks for the fast responses! I've visualized them, and I'm trying to figure out how to interpret the images. It looks like the brake is almost always set to True which seems to be the behavior across all of the routes that causes a near 0% RouteCompletion. The below DEBUG images are from deploying image_agent on routes_devtest/route_01 and routes_devtest/route_02, which are respectively deployed in Town3 and Town4.

from 2020_carla_challenge.

can you make sure to disable these black squares? i forget the flag for this but i believe it is DEBUG_CHALLENGE = 0

from 2020_carla_challenge.

Setting DEBUG_CHALLENGE=0 removed the black squares and seems to have resolved the perpetual braking issue, with the agent hitting a 17% RouteCompletion rate on the route I was running (at which point I stopped the run)

After thinking it through, I guess those black squares were present in the image that's passed to the model (I had thought they were only visible to the CARLA spectator or were drawn post model-inference). Seems like an obvious observation in retrospect...

from 2020_carla_challenge.