aiasd / adfuzz Goto Github PK

Hi!
I am able to get ADFuzz up and running, and I do see crashes! But I have some questions!

Create Apollo Master in Vehicles

SVL does not have a default "Apollo Master" for "Lincoln2017MKZ" under "Vehicles". To create one, one can duplicate "Apollo 5.0" and then add sensors "Clock Sensor" and "Signal Sensor" from "Apollo 6.0 (modular testing)".
Other preparation

Need to change the field model_id in svl_specific to one's own model_id on svl web UI.

For this specific section, can you please add more details to it? I am not sure if I am doing the correct things, and here are some more detailed questions for you.

For "Create Apollo Master in Vehicles"

• is this referring to adding a "Sensor Configurations" for Lincoln2017MKZ in LGSVL? and to do so, we "Add New Configuratoin" by uploading the configuration for both Apollo 5.0 (full analysis) and Apollo 6.0 (modular testing). In other words, combining 2 configurations?

For "Other prepartion"

• where specifically would we be modifying model_id? Is it at line 529 where we would write model_id="ID_OF_NEW_CONFIGURATION_CREATED_ABOVE"?
• After updaing svl_specific.py, it looks like we also need to update simulation_utils.py because line 115-140 will consider the new ID as unknown. Is this correct?

Thank you in advance for your response!

I have see this issue a couple times previously and I am not sure if you have seen this type of issue before, but it is the same every time on my machine.

In Apollo's dreamview the car has not crossed the intersection yet, but in LGSVL the car is way farther than what Apollo shows.
Is there some additional configuration that I need to be aware of?

Commands Used:

python ga_fuzzing.py --simulator svl --n_gen 1 --pop_size=1 --algorithm_name nsga2-un --rank_mode adv_nn --ego_car_model apollo_6 --route_type 'BorregasAve_left' --scenario_type 'turn_left_one_ped_and_one_vehicle'

Code changed:

            model_id="2e9095fa-c9b9-4f3f-8d7d-65fa2bb03921" # Apollo 6.0 (modular testing)
            ego = sim.add_agent(model_id, lgsvl.AgentType.EGO, state)
            ego.connect_bridge(BRIDGE_HOST, BRIDGE_PORT)
            ego.on_collision(on_collision)

with the most recent Apollo and ADFuzz.

Hello @AIasd , I am doing an automated testing of scenarios in CARLA simulator (CARLA 0.9.13) and the scenarios are in openscenario format. For example, I have a lane change scenario which I have written in python script and converting that to openscenario and running in CARLA with the help of scenario runner module. So is there any possibility that I can use your AD fuzzer implementation to find the parameters where my scenario fails?

Hello,

In CARLA_challenge (with LBC), It is stated that CARLA 0.9.10.1 is supported.
So, I want to know if ADFuzz supports CARLA 0.9.10.1.

Thank you.

In the instruction for "STACK2: SVL2021.3+Apollo Master" it says "put generate_route.sh inside apollo".

Where is generate_route.sh?

python ga_fuzzing.py -p 2015 --n_gen 2 --pop_size 2 -r 'town07_front_0' -c 'go_straight_town07' --algorithm_name nsga2-un --has_run_num 4 --objective_weights -1 1 1 0 0 0 0 0 0 0 --check_unique_coeff 0 0.1 0.5 --record_every_n_step 5 --debug 0

while running the above command, getting the error as shown below:
raceback (most recent call last):
File "/home/adas/anaconda3/lib/python3.8/site-packages/importlib_metadata/init.py", line 381, in from_name
return next(cls.discover(name=name))
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "ga_fuzzing.py", line 33, in
if version('carla') != '0.9.9':
File "/home/adas/anaconda3/lib/python3.8/site-packages/importlib_metadata/init.py", line 832, in version
return distribution(distribution_name).version
File "/home/adas/anaconda3/lib/python3.8/site-packages/importlib_metadata/init.py", line 805, in distribution
return Distribution.from_name(distribution_name)
File "/home/adas/anaconda3/lib/python3.8/site-packages/importlib_metadata/init.py", line 383, in from_name
raise PackageNotFoundError(name)
importlib_metadata.PackageNotFoundError: No package metadata was found for carla

Any solution for the same

Hello again! Thanks for answering all my previous questions!

I see there are many more command-line arguments and I assume the default ones are the best.

There are a few that I did not understand and did not see which part explains them.

1. has_run_num
2. objective_weights
3. check_unique_coeff

Can you help me understand what those are for?

Hello again! Hope you are doing well!
I spent some more time looking at the command line arguments and I am confused with algorithm_name

Supported algorithms are: 1. AutoFuzz (GA-UN-NN-GRAD), 2. NSGA2-SM, 3. NSGA2-DT, 4. AV-Fuzzer.
But by reading the code I see algorithm_name can be any of:

- random
- random-un
- avfuzzer
- nsga2
- nsga2-un-dt
- nsga2-dt
- nsga2-un
- naga2-emcmc
- nsga2-un-emcmc

1. Am I missing any possible input for algorithm_name?
2. Can you label supported algorithm with its corresponding algorithm_name?

Thank you in advance!

Hi,

Thanks for the efforts in creating such great tool. The results are very impressive. I am wondering if there are more details about the collision discovered in Apollo 6.0? The finding is very interesting, but the root cause of this collision seems to be unclear (both on project website and in the paper).

Thanks a lot!

Best,

Hi!

I am seeing an index out of bound error after 100 simulations. I am running ADFuzz with the following command

python ga_fuzzing.py --simulator svl --algorithm_name nsga2-un --rank_mode adv_nn --ego_car_model apollo_6 --route_type 'BorregasAve_left' --scenario_type 'turn_left_one_ped_and_one_vehicle' --n_gen 24 --pop_size 10

len(cur_X_remaining) 408
Mating finds 300 offsprings after doing 2 / 40 mating iterations

 after mating len 0 300 self.n_offsprings 300 

process_X finished
(90, 20) (100,)
Traceback (most recent call last):
  File "ga_fuzzing.py", line 1626, in <module>
    run_ga_general(fuzzing_arguments, sim_specific_arguments, fuzzing_content, run_simulation)
  File "ga_fuzzing.py", line 1515, in run_ga_general
    run_ga(fuzzing_arguments, sim_specific_arguments, fuzzing_content, run_simulation)
  File "ga_fuzzing.py", line 1459, in run_ga
    algorithm.solve()
  File "/ADFuzz/pymoo/pymoo/model/algorithm.py", line 211, in solve
    self._solve(self.problem)
  File "/ADFuzz/pymoo/pymoo/model/algorithm.py", line 291, in _solve
    self.next()
  File "/ADFuzz/pymoo/pymoo/model/algorithm.py", line 263, in next
    self._next()
  File "ga_fuzzing.py", line 1121, in _next
    self.set_off()
  File "ga_fuzzing.py", line 1023, in set_off
    clf = train_net(X_train, y_train, [], [], batch_train=200, device_name=self.device_name)
  File "/ADFuzz/fuzzing_utils/pgd_attack.py", line 741, in train_net
    for i, (x_batch, y_batch) in enumerate(train_loader):
  File "/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/ADFuzz/fuzzing_utils/pgd_attack.py", line 36, in __getitem__
    return (self.X[idx], self.y[idx])
IndexError: index 98 is out of bounds for axis 0 with size 90

Hi,

For some reason when running ADFuzz I have been seeing bridge throwing errors

Agent '9c067880-ceaa-49f5-a705-e9e6365be6ab' is missing bridge client

For the section "Other preparation" in the readme file, do we have to change every ID specified here?

    if ego_car_model == 'apollo_6_with_signal':
        model_id = '9272dd1a-793a-45b2-bff4-3a160b506d75'
    elif ego_car_model == 'apollo_6_modular':
        model_id = '2e9095fa-c9b9-4f3f-8d7d-65fa2bb03921'
    elif ego_car_model == 'apollo_6_modular_2gt':
        model_id = 'f0daed3e-4b1e-46ce-91ec-21149fa31758'
    elif ego_car_model == 'apollo_6':
        model_id = 'c354b519-ccf0-4c1c-b3cc-645ed5751bb5'
    else:
        print('ego car model is invalid:', ego_car_model)
        raise

I used to be able to run ADFuzz by using apollo_6_modular without changing anything, but could not do so today. Is it because the vehicle ID changed in LGSVL?

Hello,
I'm very grateful for your research.
And I want to use your tool, ADFuzz.
So, I want to know the specifications of your machine.
May I know which CPU and GPU you used to execute the experiment?
If you use different machines in SVL and CARLA, I wonder the specifications of each.

Thank you.

Hello,

I have a problem about your implementation of receive_zmq method in svl_script/simulation_utils.py. I find that you used sockets to receive messages. But I don't know who are the message sender, which I didn't find in this project. Can you provide any details? Thank you!

python ga_fuzzing.py -p 2021 --n_gen 15 --pop_size 50 -r 'town07_front_0' -c 'go_straight_town07' --algorithm_name nsga2-un --has_run_num 700 --objective_weights -1 1 1 0 0 0 0 0 0 0 --rank_mode adv_nn --warm_up_path --warm_up_len 500 --check_unique_coeff 0 0.1 0.5 --has_display 0 --record_every_n_step 5 --only_run_unique_cases 1 --record_every_n_step 5 --debug 0

Which path needs to be given here?

and what is its format

Hi @AIasd , hope you are doing well!

Problem Summary

I am trying to run ADFuzz with the following arguments

python3.8 ga_fuzzing.py --simulator svl \
    --n_gen 60 --pop_size 10 --algorithm_name nsga2-un --rank_mode adv_nn \
    --has_run_num 140 --objective_weights -1 1 1 0 0 0 0 0 0 0 \
    --check_unique_coeff 0 0.1 0.5 --episode_max_time 30 \
    --ego_car_model apollo_6 --route_type 'BorregasAve_left' \
    --scenario_type 'turn_left_one_ped_and_one_vehicle' \
    --record_every_n_step 5 --n_offsprings 50

• pop_size and has_run_num is configured based on #15 (comment)

• some other values are configured based on https://github.com/AIasd/ADFuzz/blob/main/doc/stack2_svl_apollo.md

• --n_gen is set to 60 because I want to generate about 600 scenarios.

However, after 100 scenarios, main will exit with the following error message

  File "/home/yuqi/ResearchWorkspace/Reproduction/ADFuzz/pymoo/pymoo/model/algorithm.py", line 291, in _solve
    self.next()
  File "/home/yuqi/ResearchWorkspace/Reproduction/ADFuzz/pymoo/pymoo/model/algorithm.py", line 263, in next
    self._next()
  File "ga_fuzzing.py", line 920, in _next
    self.set_off()
  File "ga_fuzzing.py", line 822, in set_off
    clf = train_net(X_train, y_train, [], [], batch_train=200, device_name=self.device_name)
  File "/home/yuqi/ResearchWorkspace/Reproduction/ADFuzz/fuzzing_utils/pgd_attack.py", line 742, in train_net
    for i, (x_batch, y_batch) in enumerate(train_loader):
  File "/home/yuqi/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 628, in __next__
    data = self._next_data()
  File "/home/yuqi/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 671, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/yuqi/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yuqi/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 58, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/yuqi/ResearchWorkspace/Reproduction/ADFuzz/fuzzing_utils/pgd_attack.py", line 36, in __getitem__
    return (self.X[idx], self.y[idx])
IndexError: index 94 is out of bounds for axis 0 with size 90

The issue seem to be identical to #11 which I reported a while back. I recall not seeing the error right after you fixed it, but now the error is reappearing. I ran the experiment 3 times and each time only 100 scenarios are generated.

Questions:

1. Can you take a look at the bash command and confirm whether this is the right command to run ADFuzz in attempt to reproduce the experiment discussed in the paper?
2. Will you be able to look into the issue and fix it in the near future?

Hello.
I have questions while experimenting with generating several scenarios using ADFuzz.

As far as I understand, customized_bounds_and_distributions in the scene_configs.py file, will specify the constraints of the scenario. If so, can I create the scenario situation that I want by modifying the file?

In this case, how does ADFuzz set the location of other vehicles(not ego-vehicle), obstacles, or pedestrians?
(I wonder how constraints such as "pedestrian_x_min_0" make the position of the object.)

And can I set it to an absolute position using OpenDRIVE's essential x/y/z coordinate system? (similar to setting customized_routes in scene_configs.py)

Thank you!

Hello, thank you for making such a wonderful tool.

I was able to use this tool to run fuzzing and get the following data.

I would like to know if I can see this scenario in CARLA simulator with this data.

If possible, how can I do this?

Hi! Thank you for all the responses to earlier issues. I am able to successfully run ADFuzz without any issue.
I am trying to run ADFuzz in the same setup as described in the paper at https://arxiv.org/pdf/2109.06126.pdf, 140 simulations, and 100 warm-up simulations. Are the following the command arguments to use? Can I use the default value for all other arguments?

-a nsga2-un --rank_mode adv_nn --warm_up_len 100

Hi,
For the line referenced above, you might need to add "svl_script.object_types" to import from the correct file.
from svl_script.object_types import static_types, pedestrian_types, vehicle_types

Otherwise it will show a "No Module Named 'carla'" error.

The import is correct in other files.

Thank you for sharing the project : ). I tried running the project based on CARLA (i.e., https://github.com/AIasd/ADFuzz/blob/main/doc/stack1_carla_lbc.md) and there was some confusion about some of the parameters, such as --warm_up_path , firstly I didn't understand the parameter, and secondly, It seems that the folder (e.g., path-to-warm-up-run-folder) does not exist in the project. Can you help explain this parameter and provide the folder (the folder contains the required files).

Hello again!
I have been running ADFuzz for a longer time and started seeing collisions with other vehicles.

However, for function on_collision (https://github.com/AIasd/ADFuzz/blob/main/svl_script/simulation_utils.py#L62), the function raises an exception when agent2 is None, because line 81&82 does not check if agent2 is None.

I have watched the scenario and the ego vehicle collided with a truck.

Do you know if this is another LGSVL bug like the one yesterday?

Hi! Hope you are doing well!

I have been watching the scenarios in Dreamview when ADFuzz is running and I never see pedestrian being recognized in Apollo.

I am running Apollo 6.0 with the following command

python ga_fuzzing.py --simulator svl --n_gen 1 --pop_size=1 --algorithm_name nsga2-un --rank_mode adv_nn --ego_car_model apollo_6_modular --route_type 'BorregasAve_left' --scenario_type 'turn_left_one_ped_and_one_vehicle'

Is there some setting in Apollo that I need to change to make pedestrian show up in Dreamview? I played some random LGSVL scenario and I see pedestrians being recognized.

Hi, at 99's simulation, I saw this error at line

Out of curiosity, do you regularly run ADFuzz with apollo, and are these issues only happening on my machine but not on yours?

Traceback:

Traceback (most recent call last):
  File "ga_fuzzing.py", line 1732, in <module>
    run_ga_general(fuzzing_arguments, sim_specific_arguments, fuzzing_content, run_simulation)
  File "ga_fuzzing.py", line 1611, in run_ga_general
    run_ga(fuzzing_arguments, sim_specific_arguments, fuzzing_content, run_simulation)
  File "ga_fuzzing.py", line 1555, in run_ga
    algorithm.solve()
  File "/ADFuzz/pymoo/pymoo/model/algorithm.py", line 211, in solve
    self._solve(self.problem)
  File "/ADFuzz/pymoo/pymoo/model/algorithm.py", line 291, in _solve
    self.next()
  File "/ADFuzz/pymoo/pymoo/model/algorithm.py", line 263, in next
    self._next()
  File "ga_fuzzing.py", line 1208, in _next
    self.set_off()
  File "ga_fuzzing.py", line 1023, in set_off
    X_train, X_test, xl, xu, labels_used, standardize, one_hot_fields_len, param_for_recover_and_decode = process_X(initial_X, self.problem.labels, self.problem.xl, self.problem.xu, cutoff, cutoff_end, partial, len(self.problem.interested_unique_bugs), self.problem.fuzzing_content.keywords_dict, standardize_prev=standardize_prev)
  File "/ADFuzz/customized_utils.py", line 682, in process_X
    kept_fields_non_encode = kept_fields - one_hot_fields_len
TypeError: unsupported operand type(s) for -: 'list' and 'int'