Add topic publish and json output for results visualisation

What

As explained in this issue, the interface of traffic light message in autoware will be updated.

I'm working in this branch.

tier4/autoware_perception_evaluation#15

A 2D perception evaluation function has been added.
Allow existing PERCEPTION functionality to work in the PR branch.

implement

tier4/autoware_perception_evaluation#15

Currently, there are multiple required nodes in the obstacle segmentation and the exit status is 1.
https://github.com/hayato-m126/launch_exit_status

Scenario simulator v2 avoids this problem by using ShutdownOnce instead of Shutdown

https://github.com/tier4/scenario_simulator_v2/blob/master/test_runner/scenario_test_runner/scenario_test_runner/shutdown_once.py

Description

In the current implementation, if the ground truth is zero and the number of recognized results is also zero, it is judged as a failure.
https://github.com/tier4/driving_log_replayer/blob/develop/driving_log_replayer/driving_log_replayer/criteria/perception.py#L178-L179

Expected behavior

When the ground truth is zero, return SUCCESS if the number of recognized results is also zero.

According to the installation docs there is a dlr executable available. However a pip package don't have such file:

pip install dlr
Collecting dlr
  Downloading dlr-1.10.0-py3-none-any.whl (18 kB)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from dlr) (1.26.4)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from dlr) (2.31.0)
Requirement already satisfied: distro in /usr/lib/python3/dist-packages (from dlr) (1.7.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->dlr) (3.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->dlr) (2024.2.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->dlr) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->dlr) (2.2.0)
Installing collected packages: dlr
Successfully installed dlr-1.10.0
gigi@e5806e681675:~/workspace/autoware$ dlr simulation run -p perception  -l "play_rate:=0.5"
bash: dlr: command not found

How can I install it?

Fix so that different colors are applied to tp_gt, tp_est, fp and fn

/autoware/sensing/node_alive_monitoring/topic_status/ad_service_state_monitor: sensing_topic_status

↓

/autoware/perception/node_alive_monitoring/topic_status/topic_state_monitor_obstacle_segmentation_pointcloud: perception_topic_status

Currently, only OBSTACLE_SEGMENTATION is supported.
Reorganize so that the same mechanism can be used for analysis of other modules.
Add documentation

fixes a problem that caused success even when the initial position did not fit and was not evaluated.

initial value of result must be false

test case 1 / 1 : use case: sample
--------------------------------------------------
TestResult: Passed
Passed: Convergence (Passed): NotTested, Reliability (Passed): NotTested, NDT Availability (Passed): NDT available

Use the same documentation tools as Autoware.

https://github.com/tier4/autoware-documentation

Even when localization is false, the evaluation node waits for map fit services to start, resulting in infinite wait times.

tier4/autoware_perception_evaluation#78

update for

autowarefoundation/autoware.universe#2724

https://github.com/tier4/driving_log_replayer/blob/develop/driving_log_replayer/scripts/localization_evaluator_node.py#L347-L351

Converged condition is no longer necessary because of the PR below.
autowarefoundation/autoware.universe#1873

In the current implementation, /initialpose is published for initial positioning, but change to use the service with ad-api.

https://autowarefoundation.github.io/autoware-documentation/main/design/autoware-interfaces/ad-api/list/api/localization/

The current implementation only outputs evaluation results for each dataset, even if multiple datasets are described in the perception.
In database evaluation, we would like to output the results of multiple datasets together.

The trained machine learning models used in PERCEPTION are specified in the launch file, which specifies the output directory and file name.

https://github.com/autowarefoundation/autoware.universe/blob/main/launch/tier4_perception_launch/launch/object_recognition/detection/lidar_based_detection.launch.xml#L14-L15

The current implementation checks in lidar_centrepoint direcotry, and if a different directory is specified, an infinite wait occurs.
So, delete the file conversion wait in launch as if the conversion has already been done in advance.

ros2/rosbag2#994

To be implemented after galactic support ends.

https://github.com/tier4/driving_log_replayer/actions/runs/6428222697

The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Fix runtime error when parsed value from a scenario should be a float but the actual value is an integer

autowarefoundation/autoware.universe#6556

https://ubuntu.com/blog/ros2-launch-required-nodes

This avoids lidar_centerpoint errors

tier4/autoware_perception_evaluation#66

obstacle_segmentation is a 2-node configuration in C++ and python.
Use lanelet2_extesion_python to make it a one-node configuration of python.

autowarefoundation/autoware_common#221

add except TransformException

sample code
https://docs.ros.org/en/rolling/Tutorials/Intermediate/Tf2/Writing-A-Tf2-Listener-Py.html

/planning/scenario_planning/status/stop_reasons
↓
/awapi/autoware/get/status --field stop_reason

need to launch deprecated api

autowarefoundation/autoware.universe#3023

• TP/FP/FNオブジェクトがすべて無い場合に，Successと出てしまう

• この場合，autoware_perception_evaluationにおけるPerceptionFrameResult.pass_fail_resultのTP/FP/FNオブジェクトが無くて解析処理ができないが，failが0のため，Successと出てしまう

• 理想的には，pickleを出力する際に，TP/FP/FNオブジェクトがすべて無い場合は，エラーを吐いてほしい（configの設定が適切でないと考えられるため）

• Success is displayed when all TP/FP/FN objects are missing

• In this case, the TP/FP/FN objects in PerceptionFrameResult.pass_fail_result in autoware_perception_evaluation are missing and cannot be analyzed, but the fail is 0, so Success is printed.

• Ideally, when outputting a pickle, if all TP/FP/FN objects are missing, an error should be thrown (because the config settings may not be appropriate).

tier4/autoware_perception_evaluation#7

tier4/autoware_perception_evaluation#64

Add exception handling when doing lookup transform

https://github.com/ros-tooling/rosbag2_storage_mcap

https://github.com/tier4/driving_log_replayer/tree/mcap

Correct z-coordinate with map_height_fitter when sending initial pose via api

autowarefoundation/autoware.universe#4128

To compare the timestamp of the point cloud with the timestamp of the bounding box, record the respective timestamps in result.jsonl.

Translate Japanese documents under docs into English

Erase the options that were included to address the following issue problems

autowarefoundation/autoware.universe#1982

What

Currently, driving_log_replayer saves evaluation results with pickle format in result_archive, but it does not contain version information of perception_eval.
Therefore, it sometimes causes an error by version mismatch when users want to use PerceptionAnalyzer3D.add_from_pkl(...).

Then, I want to dump python objects into pickle containing version information of perception_eval with perception_eval.util.dump_to_pkl(...) function.
This function serializes objects as dict, whose keys and values are {'version': str, 'data': Any}.
It allows users to notice the mismatching the version of perception_eval between they are using and they used to evaluate.
You can deserialize pkl with perception_eval.util.load_pkl(...).

As a rule of releases on perception_eval, I'm planning the following policy.

• Major updates : Breaking changes or directory tree updates
• Minor updates : Updates including incompatible python objects in pickle
• Micro updates : Otherwise, or tiny changes

I'm working on this in the branch shown as below

• driving_log_replayer : https://github.com/tier4/driving_log_replayer/tree/feat/dump-pkl-with-version
• perception_eval : https://github.com/tier4/autoware_perception_evaluation/tree/feat/load-pkl

Porting features developed in private repositories

use perception_online_evalutor
autowarefoundation/autoware.universe#6493

There are values, NaN and Infinity, that are not allowed in json in jsonl in perception.

use simplejson and set ignore_nan
https://stackoverflow.com/questions/6601812/sending-nan-in-json

In the current implementation, the pose passed from the scenario is thrown directly to the intiialpose service.
Since initialpose is supposed to be thrown with 2D esitmate pose and z=0, it is assumed to be used with z correction.

https://github.com/autowarefoundation/autoware.universe/blob/main/system/default_ad_api_helpers/ad_api_adaptors/src/initial_pose_adaptor.cpp#L49-L59

Hello everybody,

I followed the installation guide to set up the perception evaluation and used the sample_dataset to check the evaluation, as described in Quick Start/ Perception Evaluation.

The results of the simulation are always like this:

$ driving_log_replayer simulation run -p perception --rate 0.5
.
.
.
test case 1 / 1 : use case: sample_dataset

TestResult: Failed
Failed: 6 / 19 -> 31.58%

The number of executed tests varies between "NoData" and ~ 50. But the result is "Failed" every time. It seems like the point cloud only refreshes sporadically in rviz2.
The /sensing/lidar/concatenated/pointcloud Topic has an average rate of 0.3-0.4 Hz while testing.

The workload of the used PC is not a problem.
Am I doing something wrong?

Thank you!

https://github.com/tier4/driving_log_replayer/blob/develop/driving_log_replayer/scripts/localization_evaluator_node.py#L346

https://en.wikipedia.org/wiki/Shoelace_formula

If a bounding box exists, the point cloud is judged NG if it does not appear in the bounding box.
Annotated, but distant objects may not be problematic even if they are not visible.

Therefore, it should be possible to set from the scenario whether the bounding box is enabled in the evaluation or not, based on time

support test modes below and switch the test mode by the scenario

• detection only
• non_detection only
• detection and non_detection (Currently implemented)