In the current implementation, only timer callbacks and subscription callbacks are available.
Make it possible to measure the service.

Checklist

• I've read the contribution guidelines.
• I've confirmed the issue is certainly related to this repository. If not, please consider to make a new issue in https://github.com/tier4/caret/issues

Description

Hi, I followed the tutorial. https://tier4.github.io/CARET_doc/main/tutorials/visualization/
"Visualizing latency of node chain with message flow" where the following error occurs.

$ from caret_analyze.plot import Plot

Succeed to find record_cpp_impl. the C++ version will be used.
Traceback (most recent call last):
File "", line 1, in
File "/home/caret/ros2_caret_ws/build/caret_analyze/caret_analyze/plot/init.py", line 17, in
from .plot_facade import Plot
File "/home/caret/ros2_caret_ws/build/caret_analyze/caret_analyze/plot/plot_facade.py", line 21, in
from .callback_scheduling import CallbackSchedulingPlot, CallbackSchedulingPlotFactory
File "/home/caret/ros2_caret_ws/build/caret_analyze/caret_analyze/plot/callback_scheduling/init.py", line 15, in
from .callback_scheduling_plot import CallbackSchedulingPlot
File "/home/caret/ros2_caret_ws/build/caret_analyze/caret_analyze/plot/callback_scheduling/callback_scheduling_plot.py", line 20, in
from bokeh.plotting import Figure, save, show
ImportError: cannot import name 'Figure' from 'bokeh.plotting' (/home/caret/.local/lib/python3.10/site-packages/bokeh/plotting/init.py)**

How do you solve it?

Hello, does caret support ROS2 application tracing for multiple computing devices. For example, different ros2 nodes run on different computing devices.

The calculation of to_records is occurring every time it is visualized, and it is taking a lot of time.
The result of to_records should not change and should be cached.

Support the output of graphs as described in the following link.

https://discourse.ros.org/t/consideration-of-performance-measurement-methods/22453

Checklist

• I've read the contribution guidelines.
• I've confirmed the issue is certainly related to this repository. If not, please consider to make a new issue in https://github.com/tier4/caret/issues

Description

hi, when i am analyzing data on another machine. I have encountered the following error

Checklist

• I've read the contribution guidelines.
• I've confirmed the issue is certainly related to this repository. If not, please consider to make a new issue in https://github.com/tier4/caret/issues

Description

hi, i record data form one machine. and analyze data on another machine。
when i run ros2 caret check_ctf -d xx, i met the follow error.
do you have any idea?

Hello.

• I'm doing work on profiling autoware.universe. At present, I have encountered the problem of high end-to-end delay, and I would like to ask you for advice.

    The machine configuration is as follows:
    Scenario: https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/
    Environment
            OS: Ubuntu 20.04, 5.15.0-52-generic
            ROS: ROS2 Galactic
            CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz 40 Core
            GPU: Tesla T4
            Memory: 64GByte
            Storage: 180GByte
  

• When using the Caret tool to analyze the delay, it is found that the delay is much higher than 100ms, and the bottleneck is not in the calculation of a certain ROS node. Every part seems to have a relatively high delay, such as attachments.

test.pdf

• When running autoware, the average CPU utilization rate is 40%, the highest load is 20, and the highest IO TPS is 190. These indicators do not show the bottleneck of the program.

• I have the following questions:

Do you know the bottleneck that restricts the high latency of the system?

Checklist

• I've read the contribution guidelines.
• I've confirmed the issue is certainly related to this repository. If not, please consider to make a new issue in https://github.com/tier4/caret/issues

Description

read_arch.get_path('<target_path>').verify() returns True, so Caret_analysze confirmed that it should understand the whole path.
In my scenario it doesnt show the whole path.

How it is:

How it should be like:

Here you can find my setup (notebook, architecturefile and trace data): https://ufile.io/p5ua9ble

I found typo.

https://github.com/tier4/CARET_analyze/blob/2f53c7506baacd669cfc2c7cab66152821d8b9bb/caret_analyze/value_objects/path.py#L144

The following error occurs when attempting to obtain trace data for a very short duration (e.g., 1 second) by filtering.

Traceback (most recent call last):
  File "/opt/ros/galactic/bin/ros2", line 11, in <module>
    load_entry_point('ros2cli==0.13.3', 'console_scripts', 'ros2')()
  File "/opt/ros/galactic/lib/python3.8/site-packages/ros2cli/cli.py", line 67, in main
    rc = extension.main(parser=parser, args=args)
  File "/home/saitamaemb4/ros2_caret_ws/build/ros2caret/ros2caret/command/caret.py", line 32, in main
    return extension.main(args=args)
  File "/home/saitamaemb4/ros2_caret_ws/build/ros2caret/ros2caret/verb/node_summary.py", line 79, in main
    tmp_lttng = Lttng(args.trace_dir, event_filters=[
  File "/home/saitamaemb4/ros2_caret_ws/build/caret_analyze/caret_analyze/infra/lttng/lttng.py", line 187, in __init__
    self._counter = EventCounter(data, validate=validate)
  File "/home/saitamaemb4/ros2_caret_ws/build/caret_analyze/caret_analyze/infra/lttng/event_counter.py", line 34, in __init__
    self._validate()
  File "/home/saitamaemb4/ros2_caret_ws/build/caret_analyze/caret_analyze/infra/lttng/event_counter.py", line 85, in _validate
    raise InvalidTraceFormatError(
caret_analyze.exceptions.InvalidTraceFormatError: Failed to found trace point added by forked rclcpp. Measurement results will not be correct. The binary may have been compiled without using fork-rclcpp.

The measurement result is likely to be the first maximum value calculated.
To ignore transient maximum values caused by caching and other effects, add transient_time as an argument.

When we checked the operation, we found that the measurement was not done as expected.

On_data_available → callback execution was assumed, but there was a case that the trace point of on_data_available was not output.
Need to check.

If the horizontal axis is the system time, multiple results cannot be compared.
Support the output of a graph with the horizontal axis being the time in /clock.

The time between callback_end and callback_start may appear to go back.
This makes it difficult to understand, so make it possible to switch between them with an argument.

Hello.
thank you for your contribution to caret.
In the process of using, I encountered some problems. Please help me to solve them.
The environment I use is as follows.

ROS Galactic
Ubuntu 20.04
LTTng stable-2.12
Linux Kernel 5.15.x
Python3 3.8

CARET is installed according to the document [https://tier4.github.io/CARET_doc/refs-tags-v0.2.3/.](https://tier4.github.io/CARET_doc/refs-tags-v0.2.3/)

CARET_ Demos can be used normally.
However, I encountered some problems when using autoware.universe.

I follow the example in https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/.
Open one terminal.
ros2 trace -s plan_sim -k -u "ros2*"

Open a new terminal.
ros2 launch autoware_launch logging_simulator.launch.xml map_path:=$HOME/autoware_map/sample-map-rosbag vehicle_model:=sample_vehicle sensor_model:=sample_sensor_kit

Open a new terminal.
ros2 bag play ~/autoware_map/sample-rosbag/sample.db3 -r 0.2.

Using the following command, the following problems occur.

ros2 caret check_ctf -d ./plan_sim/ =
Succeed to find record_cpp_impl. the C++ version will be used.
763096 events found.
loading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 763096/763096 [00:33<00:00, 23096.29it/s]
WARNING : 2022-09-30 14:39:55 | Multiple executors using the same callback group were detected.The last executor will be used. executor address: [94228999023792, 140722136593168]
763096 events found.
loading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 763096/763096 [00:03<00:00, 247700.66it/s]
WARNING : 2022-09-30 14:40:13 | Multiple executors using the same callback group were detected.The last executor will be used. executor address: [94228999023792, 140722136593168]
/home/inspur/lhg/ros2_caret_ws/build/caret_analyze/caret_analyze/infra/lttng/lttng_info.py:436: FutureWarning: In a future version of pandas, a length 1 tuple will be returned when iterating over a groupby with a grouper equal to a list of length 1. Don't supply a list with a single grouper to avoid this warning.
WARNING : 2022-09-30 14:40:33 | Failed to identify subscription. Several candidates were found. node_name: /system/emergency_handler, topic_name: /system/emergency/hazard_status
WARNING : 2022-09-30 14:40:33 | Failed to identify subscription. Several candidates were found. node_name: /system/emergency_handler, topic_name: /system/emergency/hazard_status
WARNING : 2022-09-30 14:40:34 | Failed to identify subscription. Several candidates were found. node_name: /transform_listener_impl_5556962b5700, topic_name: /tf_static
WARNING : 2022-09-30 14:40:34 | Failed to identify subscription. Several candidates were found. node_name: /transform_listener_impl_5556962b5700, topic_name: /tf
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_93830345088256
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_93864174799728
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_93881056423552
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94142363909952
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140428317844960
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94142371906640
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94270342456656
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94270343180896
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94331374011344
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139738505894400
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94391240707024
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140366512405792
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140366512454928
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140366512528112
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140366512569984
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140366512667024
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140366512969056
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140365645901904
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140365646087840
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140365646269200
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140365646391504
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140365646417712
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140365646773312
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94418804172576
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140141226138624
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94418804807088
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94418805404000
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140142635453616
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94529662460976
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94605562524448
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139922585626944
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139923457994160
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139921511814912
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139921847278928
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139921847489152
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139923793458896
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139921579058160
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139924330421952
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139923726617984
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139922921211744
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139922182817440
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139923793627104
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139921847969424
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139924330785248
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_139923793626912
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94728999438448
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94796218103888
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94796218245056
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_140152970436320
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94884374091280
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94519088361248
WARNING : 2022-09-30 14:40:34 | Failed to find callback group. callback_group_id=callback_group_94624423775440
WARNING : 2022-09-30 14:40:34 | Failed to identify subscription. Several candidates were found. Skip loading.node_name: /planning/scenario_planning/lane_driving/motion_planning/surround_obstacle_checker, topic_name: /localization/kinematic_state
WARNING : 2022-09-30 14:40:34 | Failed to identify subscription. Several candidates were found. Skip loading.node_name: /planning/scenario_planning/lane_driving/motion_planning/surround_obstacle_checker, topic_name: /localization/kinematic_state
WARNING : 2022-09-30 14:40:34 | Failed to identify subscription. Several candidates were found. Skip loading.node_name: /system/emergency_handler, topic_name: /system/emergency/hazard_status
WARNING : 2022-09-30 14:40:34 | Failed to identify subscription. Several candidates were found. Skip loading.node_name: /system/emergency_handler, topic_name: /system/emergency/hazard_status
WARNING : 2022-09-30 14:40:34 | Duplicate parameter callback found. node_name: /control/vehicle_cmd_gate, callback_type: timer_callbackperiod_ns: 100000000
WARNING : 2022-09-30 14:40:34 | Duplicate parameter callback found. node_name: /system/system_monitor/hdd_monitor, callback_type: timer_callbackperiod_ns: 1000000000
WARNING : 2022-09-30 14:40:34 | Duplicate parameter callback found. node_name: /system/system_monitor/process_monitor, callback_type: timer_callbackperiod_ns: 10000000

InvalidArgumentError raises when CppImpl is disabled.

File ~/ros2_caret_ws/src/CARET/CARET_analyze/caret_analyze/infra/lttng/records_source.py:523, in RecordsSource.intra_proc_comm_records(self)
    502 """
    503 Compose intra process communication records.
    504 
   (...)
    520 
    521 """
    522 sink_records = self._data.dispatch_intra_process_subscription_callback_instances
--> 523 intra_publish_records = merge_sequencial_for_addr_track(
    524     source_records=self._data.rclcpp_intra_publish_instances,
    525     copy_records=self._data.message_construct_instances,
    526     sink_records=sink_records,
    527     source_stamp_key=COLUMN_NAME.RCLCPP_INTRA_PUBLISH_TIMESTAMP,
    528     source_key=COLUMN_NAME.MESSAGE,
    529     copy_stamp_key=COLUMN_NAME.MESSAGE_CONSTRUCT_TIMESTAMP,
    530     copy_from_key=COLUMN_NAME.ORIGINAL_MESSAGE,
    531     copy_to_key=COLUMN_NAME.CONSTRUCTED_MESSAGE,
    532     sink_stamp_key=COLUMN_NAME.DISPATCH_INTRA_PROCESS_SUBSCRIPTION_CALLBACK_TIMESTAMP,
    533     sink_from_key=COLUMN_NAME.MESSAGE,
    534     columns=[
    535         # 'tid', needed.
    536         COLUMN_NAME.DISPATCH_INTRA_PROCESS_SUBSCRIPTION_CALLBACK_TIMESTAMP,
    537         COLUMN_NAME.PUBLISHER_HANDLE,
    538         COLUMN_NAME.CALLBACK_OBJECT,
    539         COLUMN_NAME.RCLCPP_INTRA_PUBLISH_TIMESTAMP,
    540         COLUMN_NAME.MESSAGE_TIMESTAMP,
    541     ],
    542     progress_label='bindig: publish_timestamp and message_addr',
    543 )

InvalidArgumentError: Column names do not match. tid, 

Checklist

• I've read the contribution guidelines.
• I've confirmed the issue is certainly related to this repository. If not, please consider to make a new issue in https://github.com/tier4/caret/issues

Description

hi，i record data on ubuntu20.04. and analyze data on ubuntu22.04. i open the jupyter-lab and run this scripts

After execution, it redirected to a webpage

Make it possible to get callbacks to publish automatically.
If we restrict the callbacks in a node to be exclusive, we can determine the callbacks to publish chronologically.

# event, timestamp, tid, tracepoint arguments
callback_start, 0s, 100, callback_A_ptr
rclcpp_publish, 0.1s, 100
callback_end, 0.3s, 100

It cannot be measured correctly in complex cases where messages are put into buffers.
Use the value of header timestamp and add a case to track the message.

This area needs to be corrected.

https://github.com/tier4/CARET_analyze/blob/7585218295a527817c23b48ce794bab3b1b59fc5/caret_analyze/communication.py#L105