ri-se / smirk Goto Github PK

Suggested new formulation: "SYS-PER-REQ4: In any sequence of 5 consecutive frames from a video feed any pedestrian shall not be missed in more than 20% of the frames."

The running text shall state 100 scenarios in total, not 200.

The manuscript has been accepted for publication in Software Quality Journal. Update citation information accordingly and increment the SMIRK version to v1.0.

The new pedestrian and cylinder should be listed in the introduction of the Data Management Specification.

Add another negative class [N5] with out-of-distribution (OOD) examples, i.e., the cylinder shape in the ESI Pro-SiVIC object catalog. This class should be added to the development data to make sure we have negative examples for the training of the variational autoencoder used for OOD detection.

It is confusing for the reader that FP means different things for SMIRK on the system level and for the model's bounding boxes on a lower level. We shall rename it on this level to avoid confusion, it has no very limited impact on the rest of the documentation. New proposed names in Section 3.2: FN = Missed pedestrian, FP = Ghost braking

The statistical analysis of the data sets was done using Great Expectations. The DMS shall be updated to describe the process.

According to Barredo Arrieta (2020):

"interpretability refers to a passive characteristic of a model referring to the level at which a given model makes sense for a human observer. /.../ By contrast, explainability can be viewed as an active characteristic of a model, denoting any action or procedure taken by a model with the intent of clarifying or detailing its internal functions"

This would make more sense for SMIRK.

• Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, Francisco Herrera, Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Information Fusion, Volume 58, 2020, pp. 82-115,

The 50 m threshold for pedestrian detection and recognition in the performance requirements does not match the ODD. If ego car moves at 70 km/h, a TTC of 4 s means that the pedestrian can be almost 80 m away.

Check https://github.com/fcakyon/yolov5-pip

It has 250k+ total downloads from PYPI: https://pepy.tech/project/yolov5

Since true negatives can't be calculated for object detection we can't use False Positive Rate as a metric.

The most similar metric that can be calculated is False Positives Per Image (FPPI)

FPPI = False Positives / Number of images

FPPI is used as a metric for pedestrian detection in e.g.

• Caltech Pedestrian Detection Benchmark
• Assuring the Safety of Machine Learning for Pedestrian Detection at Crossings | SpringerLink

May I use another open source simulator to build a research enviroment, like Carla? if Yes, could you please provide what folder shoule be cared?

Two additional pedestrian models have been added to the dataset:

• Business casual female
• Casual male

Previous casual male has been renamed to business casual male.

The new naming of positive examples is as follows:

• [P1] Casual female
• [P2] Casual male
• [P3] Business casual female
• [P4] Business casual male
• [P5] Business female
• [P6] Business male
• [P7] Child
• [P8] Male construction worker

The data split is updated to:

• Development [P2][P3][P6][N5] (see #15)
• Internal Test [P1][P4][N1][N2]
• Verification [P5][P7][P8][N3][N4]

"In any sequence of 5 consecutive frames from a 10 FPS video feed, no pedestrian within 80 m shall be missed in more than 20% of the frames" shall be expressed using fractions. "Any sequence" is too strict, as model testing of rolling windows show that occlusion and far away pedestrians lead to misses. The requirement needs to be relaxed.

Ultralytics/yolov5 uses the copyleft GNU General Public License v3.0. As we integrate yolov5, the SMIRK source code must be relicensed accordingly.

The GSN figures showing argument patterns need to be updated as follows:

• Assurance scoping: Add IDs
• Safety reqts: Update number of robustness reqts
• ML verification: Use acronym ODD
• Model learning: Remove transfer learning

"SYS-PER-REQ1: The pedestrian recognition component shall identify pedestrians with an accuracy of 93% when they are within 50 m."

True positive rate would be a better metric than accuracy, and also align with SYS-PER-REQ2 that specifies the false negative rate.

Table 4 in Section 4.1 contains a . Replace by a range that covers the value used in the ESI Pro-SiVIC scenario, i.e., 5.6 m.

Evaluation of object detection models is non-trivial and relies on the metric intersection over union. The fact that false positives (FP) appear due to low IoU scores despite parts of a pedestrian indeed is detected is often counter-intuitive, i.e., "how can a detected pedestrian ever be an FP?" An FP means that the intersection over union between a predicted bounding box and a ground truth bounding box is less that 0.5. This does not necessarily mean that the model predicted a non-pedestrian as a reason to break - our model testing shows that this often means that a fraction of a real pedestrian was detected. Commencing emergency braking in such cases would be valid.

The corresponding performance requirement shall be set to 1%. The hazardous non-pedestrian emergency braking will be mitigated by the safety cage architecture.

SYS-SAF-REQ should be SYS-SAF-REQ1 in the first sentence.