HeLiPR Dataset sensor time synchronization about helipr-file-player HOT 12 CLOSED

Dear Kaho

Here’s a more detailed explanation, incorporating the points you're interested in:

1. Sensor Time Synchronization and Measurement Time Frames: In our approach to sensor time synchronization, it’s important to emphasize that although the measurement times from the sensors (Ouster, VLP, LIVOX, Aeva, CPT7, Xsens) are not identical, they are recorded within the same time frame reference. This method does not imply simultaneous measurement but ensures that the data are comparable and coherent when integrated. The evidence of this successful synchronization is observed through our use of LiDAR-Inertial SLAM (Simultaneous Localization and Mapping) techniques and the alignment with INS (Inertial Navigation System) ground truth data. The precise alignment of LiDAR point clouds with INS data, accounting for extrinsic parameters, validates that the various sensor measurements are effectively synchronized within the same time frame, allowing for accurate and integrated data analysis.

2. Absence of a Main Sensor Source for Synchronization: We do not designate any specific sensor as the primary clock source for synchronization purposes. Instead, synchronization across all sensors is achieved through the implementation of the Precision Time Protocol (PTP) and ROS time. This strategy ensures a universal timekeeping system that aligns all sensor data with a consistent and unified time reference, facilitating the integration and analysis of this multi-sensor dataset. Our synchronization technique is critical for the accurate application of SLAM and for ensuring the reliability of INS alignment, as it allows for the cohesive fusion of data from diverse sensors, enhancing the dataset's utility for various research applications.

I hope this response provides the detailed insights you were seeking. If there are any further aspects you’d like to explore or questions you have, please feel free to reach out.

Best wishes,

Minwoo

from helipr-file-player.

Dear minwoo,

Thank you for your reply. I am very interested in this topic and hope to continue to communicate with you.

1. Above, you mainly mentioned that you use ime Protocol (PTP) and ROS time to ensure time synchronization. May I ask what equipment do you use to output synchronous PTP signals? At the same time, ROS time here is responsible for accepting the calibration time of PTP and then transmitting the time stamp?

2. I noticed that the devices you use, such as (Livox、Ouster ...), all support PTP and PPS time synchronization. By the way, I noticed that, the Xsens only support PPS time synchronization way. Why do you consider the way of PTP time synchronization instead of the way of PPS synchronization triggering?

Looking forward for your reply~

Best wishes,
Kaho

from helipr-file-player.

Dear Kaho,

Thank you very much for your keen interest and insightful questions regarding our synchronization approach. I'm glad to continue this engaging discussion with you.

1. Regarding the Precision Time Protocol (PTP) signals, PTP operates on a network basis, allowing for time synchronization across devices connected to the same network without the need for specialized hardware. This capability is particularly advantageous in our setup, where seamless integration and flexibility are key. For more detailed information on how PTP time synchronization works, especially with Livox devices, I recommend consulting the Livox documentation, which provides a comprehensive overview: Livox PTP Time Synchronization. This approach eliminates the need for external equipment to output synchronous PTP signals, as the synchronization is managed through the network infrastructure itself.

2. Our decision not to require all sensors to measure simultaneously, but rather to ensure that measurements are taken within the same time frame for comparison, is based on the flexibility and scalability of our system. The Xsens, for instance, already supports UTC (ROS Time), negating the need to supply it with PTP signals. We only needed to provide PTP signals to specific sensors like Aeva and Avia to shift their measurement basis from local time to UTC-based measurement time. Although PPS (Pulse Per Second) synchronization offers the advantage of triggering simultaneous measurements across devices, it introduces hardware constraints and design challenges, particularly as the number of sensors increases. Therefore, we chose PTP over PPS for its ability to achieve sufficient synchronization without the aforementioned limitations of PPS, ensuring a more streamlined and adaptable system setup.

Your curiosity about our synchronization methodology is much appreciated, and I hope this clarifies our approach. PTP provides us with the necessary flexibility and ease of integration for our diverse set of sensors, including Livox and Ouster, while still maintaining accurate time synchronization critical for our research.

Looking forward to further discussions and thank you once again for your interest.

Best wishes,
Minwoo

from helipr-file-player.

Dear minwoo,
Thank you for your reply. In response to your reply, I still have a few questions to want to discuss with you~

1. As far as I know, multi-sensor PTP time synchronization, need to set the master clock source, in HeLiPR dataset, do your PTP master clock source is supported by PC? I know the use of linuxptp(ptp4l) way can achieve pc network port and other sensors time synchronization, I wonder if you also use this way?

2. When synchronizing the time of Xsens sensor, I guess you have run the ROS driver of Xsens directly. The timestamp of IMU obtained through the Ros driver of Xsens is derived from the ros-api ros::time::now(), which generally refers to the UTC time of the current system. Therefore, Xsens time is synchronized with other sensor time. In this process, I am considering whether the time is accurate enough, because there is a certain delay when the sensor starts sampling and the original data of the IMU is assigned. On the other hand, the sampling frequency of the IMU is very high (100-200HZ), which may have little impact. I hope I can reach an agreement with you on this part.

3. I remember that many inertial navigation devices do not directly support PTP time synchronization, but satellite timing of sensors or industrial computers through PPS + GPRMC. In the HeLiPR dataset, does CPT7 support PTP time synchronization directly with other sensors, or does it use additional time synchronization hardware?

Looking forward for your reply~

Best wishes,
Kaho

from helipr-file-player.

Dear Kaho,

Thank you for your continued interest and insightful queries regarding our time synchronization strategies in the HeLiPR dataset. I appreciate the depth of your questions and am happy to provide further details:

1. Regarding your inquiry about the master clock source for multi-sensor PTP time synchronization, I can confirm that synchronization through the PC network port is indeed achievable within our setup. Specifically, we employ PTP synchronization for Livox and Aeva sensors. These particular sensors do not inherently provide UTC time but instead offer local time, which necessitates our use of PTP to ensure coherent time synchronization across all devices in the network.

2. Your concerns about the potential time discrepancy arising from sensor data publication delays, especially in the context of the Xsens sensor, are understandable. However, we have determined that the slight delays do not significantly impact overall data accuracy. This is largely because the Xsens sensor operates at a high sampling rate (100-200Hz) and primarily publishes 9 degrees of freedom (9-DOF) data rather than large datasets like point clouds. Therefore, we believe the accuracy is sufficient for our applications, despite the minor time lag in sensor data assignment.

3. As for your question on the CPT7's support for direct PTP time synchronization, like the case with the Xsens sensor, the SPAN CPT-7 ROS driver makes use of UTC Time, obviating the need for additional time synchronization hardware. This approach simplifies our system's architecture while still maintaining the synchronization accuracy required for our dataset's integrity.

I hope this response addresses your queries more thoroughly. Your engagement with our work is greatly appreciated, and I look forward to any more questions you might have.

Best wishes,
Minwoo

from helipr-file-player.

Dear minwoo,
@minwoo0611

Thank you for your reply. In response to your reply, I still have a few questions to want to discuss with you~

1. I noticed that, "you use the SPAN CPT-7 ROS driver makes use of UTC Time". May I ask why not use the satellite timing of SPAN CPT-7? As far as I know, the satellite time of SPAN CPT-7 is more accurate than the UTC time of PC System, meanwhile it can avoid the interference of ROS.

2. As shown in the following figure, I drew the time synchronization framework of the HeLiPR dataset according to your reply.
   
   May I ask whether the time source of each sensor in the dataset is roughly like this?

• PTP time synchronization:
  VLP16 LiDAR, Aeva LiDAR, Ouster LiDAR, Livox LiDAR, timestamp from the PC hardware network port device (can provide PTP signal).
• ROS Drivers:
  Xsens, SPIN-CPT7, the official ros driver of the timestamp source, the general time assignment is ros::time::now(), which is actually the current PC system UTC time.

If the preceding time synchronization method is used, do you need to synchronize the time between the hardware network port and the PC system? As far as I know, PC's hardware network ports have their own clocks.

3. I noticed that HeLiPR-File-Player can easily play the HeLiPR dataset. Meanwhile there is a stamp.csv file on the dataset. I would like to ask whether this file is mainly stored and which sensor has information at each moment. Then let HeLiPR-File-Player can look for the corresponding folder according to this file, and read the sensor data with data at that time.

4. Why is the HeLiPR dataset stored in bin mode instead of ROSBAG mode? Is the original acquisition method of all sensors collected by ROS and then converted to the original data format(.bin & .csv) for storage?

Looking forward for your reply~
Best wishes,
Kaho

from helipr-file-player.

Dear Kaho,

Thank you for your patience and for your insightful questions. Here are the clarifications and responses to the points you've raised:

1. Use of UTC Time Over Satellite Time: While satellite time provided by the SPAN CPT-7 is indeed more accurate, we opted for UTC time primarily to avoid complexities associated with Pulse Per Second (PPS) synchronization. PPS synchronization can be challenging to implement on our hardware platform, and by using UTC, we can more easily aggregate all data within a common timeframe, ensuring consistency across our dataset.

2. Time Synchronization Among Sensors: You're correct that hardware network ports have their own clocks, which can be an advantage for Precision Time Protocol (PTP) time synchronization. In our setup, we indeed use PTP synchronization for Aeva and Livox Avia LiDARs, as they are connected to the same ethernet network. This approach helps us maintain precise time alignment among these sensors. However, due to potential network traffic issues and because some sensors like Ouster and Velodyne provide their UTC timestamps directly, we did not extend PTP synchronization to all devices. This decision was made to balance between time synchronization accuracy and system complexity.

3. Function of stamp.csv File: The stamp.csv file plays a crucial role in our dataset; it logs the timestamps for each sensor reading, acting as an index. This allows the HeLiPR-File-Player to efficiently locate and access the corresponding sensor data for any given moment, facilitating easier data playback and analysis. You can refer this site to get more information about stamp.csv

4. Data Storage Format (Bin vs. ROSBAG): The choice to store data in binary (bin) and CSV formats instead of ROSBAG was driven by practical considerations. While ROS is indeed utilized for initial data capture due to its ease of use and flexibility, storing the dataset directly in ROSBAG format encountered challenges, including potential frame drops and issues related to handling large volumes of data. By converting the data into bin and CSV formats at each moments, we were able to achieve a more stable and manageable data storage solution, ensuring the integrity and accessibility of the dataset for users.

I hope these answers help clarify your questions. My response has been delayed due to increased workload recently. If you have additional questions or need more information, please feel free to contact us.

Best wishes,
Minwoo

from helipr-file-player.

Dear minwoo,

Thank you very much for your detailed response;
I've learned a lot from it. I have a question about data collection for datasets: Do you use separate software for each sensor during data acquisition? I ask because you mentioned "including potential frame drops and issues related to handling large volumes of data." It seems using ROS drivers to record all sensor topics might result in large rosbag files, thus risking data loss. However, recording data with each sensor's SDK seems cumbersome.

Looking forward for your reply~

Best wishes,
Kaho

from helipr-file-player.

Dear Kaho,

I'm glad to hear that you found the information useful. Your question about our data collection methodology is insightful. Indeed, using ROS to record all sensor data simultaneously with a single command (rosbag record -a) can sometimes lead to challenges, such as potential frame drops and difficulties in managing the vast amounts of data generated by multiple sensors.

To mitigate these issues and ensure we capture data as reliably and efficiently as possible, we have developed our own sensor driver framework. This framework allows us to process data from each sensor individually and simultaneously, leveraging multithreading to handle the concurrent data streams effectively.

Here's an overview of our data acquisition pipeline:

1. Initialize ROS Sensor Drivers: We start by turning on the ROS drivers for each piece of hardware. This step ensures that each sensor is ready to transmit data.
2. Process Data with Our Custom Sensor Drivers: Next, we process the incoming data streams through our custom-developed sensor drivers. These drivers are designed to handle each sensor's data in real-time, utilizing threading to manage simultaneous data acquisition without loss.
3. Save Data in Bin and CSV Formats: Finally, the processed data is saved in both binary (.bin) and CSV (.csv) formats for each moment of data capture. This approach allows us to maintain high data integrity and accessibility for future analysis.

By adopting this custom approach, we avoid the common pitfalls associated with large-scale data collection using standard ROS tools alone for saving, ensuring that we capture a comprehensive and accurate dataset.

I hope this answers your question. Please let me know if there's anything else you'd like to know or discuss.

Best wishes,
Minwoo

from helipr-file-player.

Dear minwoo,

Thank you for your response; it was very enlightening. I would like to inquire if there are any references online for the custom sensor driver framework you mentioned. Based on my current understanding, it roughly involves the following steps:

1. Create multiple threads.
2. Each thread includes a separate ROS (Robot Operating System) subscription callback function.
3. When sensor data arrives, it is immediately stored in a queue and written to files in formats such as .txt, .bin, or .csv.
4. Based on the timestamps of all sensor data, a stamp.csv file is constructed. This allows the HeLiPR-File-Player to efficiently locate and access the corresponding sensor data for any given moment.

Here is the pseudocode I generated with the help of GPT. I find that discussing with you is a process from which I can learn a lot. Thank you very much.

queue<DataType> dataQueue; // Define a queue to store sensor data.
mutex queueMutex; // Define a mutex to protect the data queue.

void writeDataToFile(DataType data) {
    // Write data to txt, bin, or csv files without specifying file handling details.
}

void processData() {
    while (true) { // Continuously process data from the queue.
        queueMutex.lock();
        if (!dataQueue.empty()) {
            DataType data = dataQueue.front(); dataQueue.pop(); queueMutex.unlock(); writeDataToFile(data);
        } else {
            queueMutex.unlock();
        }
    }
}

void sensorCallback(DataType data) {
    // Push received sensor data into the queue.
    queueMutex.lock(); dataQueue.push(data); queueMutex.unlock();
}

int main() {
    thread processDataThread(processData); // Start the data processing thread.
    // Assume sensorCallback(data) is called elsewhere, simulating sensor data reception.
    processDataThread.join(); // Wait for the data processing thread to finish.
    return 0;
}

Best wishes,
Kaho

from helipr-file-player.

Dear Kaho,

It's great to see your progress. For the custom sensor driver framework you're working on, a practical reference you might find helpful is our file player or the ROS Message Player for pohang camnal dataset available at this GitHub repository. Although it's designed for playing back data, understanding its code can offer insights into managing sensor data and working with ROS, which is essentially the reverse of what you're aiming to achieve.

Your approach and pseudocode seem well-thought-out.

Please feel free to reach out if you have more questions or need further assistance.

Best,
Minwoo

from helipr-file-player.

Dear minwoo,

Thank you very much for your detailed response. Thank you for your great help again~

Best wishes,
Kaho

from helipr-file-player.