Orientation angle of the bounding box with respect to the camera about nuscenes-devkit HOT 10 CLOSED

Hi Lorenzo,

• As described in the sample_annotation table of the schema, every annotation is stored in global coordinates (that includes translation, rotation, velocity).

• However the method get_sample_data transforms the box into the current sensor's coordinate frame (described here). So you need to make sure that the token_sd corresponds to the right camera.

• Finally note that we store rotations using the third-party pyquaternion class. The quaternion can describe arbitrary rotations, so degrees is not necessarily the yaw angle. To convert from quaternion to yaw angle, you can use the following method (which will soon be part of the devkit):

def quaternion_yaw(q: Quaternion) -> float:
    """
    Calculate the yaw angle from a quaternion.
    See https://en.wikipedia.org/wiki/Conversion_between_quaternions_and_Euler_angles.
    :param q: Quaternion of interest.
    :return: Yaw angle in radians.
    """

    a = 2.0 * (q[0] * q[3] + q[1] * q[2])
    b = 1.0 - 2.0 * (q[2] ** 2 + q[3] ** 2)

    return np.arctan2(a, b)

Let me know if you have any more questions.

from nuscenes-devkit.

Hi.

I should have mentioned that the quaternion_yaw function only works in the lidar or global frame.
It does not work in the camera frame, which uses a different coordinate system.
Please check out this code:

from pyquaternion import Quaternion
import numpy as np
import matplotlib.pyplot as plt

from nuscenes_utils.nuscenes import NuScenes


def quaternion_yaw(q: Quaternion, in_image_frame: bool=True) -> float:

    if in_image_frame:
        v = np.dot(q.rotation_matrix, np.array([1, 0, 0]))
        yaw = -np.arctan2(v[2], v[0])
    else:
        v = np.dot(q.rotation_matrix, np.array([1, 0, 0]))
        yaw = np.arctan2(v[1], v[0])

    return yaw


nusc = NuScenes()

sd1_im = [sd for sd in nusc.sample_data if sd['filename'] == 'samples/CAM_FRONT/n015-2018-08-01-16-32-59+0800__CAM_FRONT__1533112709162460.jpg'][0]
sd2_im = [sd for sd in nusc.sample_data if sd['filename'] == 'samples/CAM_FRONT/n015-2018-07-18-11-18-34+0800__CAM_FRONT__1531884168362460.jpg'][0]

s1 = nusc.get('sample', sd1_im['sample_token'])
s2 = nusc.get('sample', sd2_im['sample_token'])

# Camera
sd1 = sd1_im
sd2 = sd2_im
_, boxes1, _ = nusc.get_sample_data(sd1['token'], box_vis_level=0)
_, boxes2, _ = nusc.get_sample_data(sd2['token'], box_vis_level=0)
box1 = boxes1[0]
box2 = boxes2[3]
yaw1 = quaternion_yaw(box1.orientation) / np.pi * 180
yaw2 = quaternion_yaw(box2.orientation) / np.pi * 180
yaw_diff = yaw1 - yaw2
print(yaw1)
print(yaw2)
print(yaw_diff)
nusc.render_sample_data(sd1['token']); plt.show()
nusc.render_sample_data(sd2['token']); plt.show()

# Lidar
sd1 = nusc.get('sample_data', s1['data']['LIDAR_TOP'])
sd2 = nusc.get('sample_data', s2['data']['LIDAR_TOP'])
_, boxes1, _ = nusc.get_sample_data(sd1['token'], box_vis_level=0)
_, boxes2, _ = nusc.get_sample_data(sd2['token'], box_vis_level=0)
box1 = boxes1[0]
box2 = boxes2[3]
yaw1 = quaternion_yaw(box1.orientation) / np.pi * 180
yaw2 = quaternion_yaw(box2.orientation) / np.pi * 180
yaw_diff = yaw1 - yaw2
print(yaw1)
print(yaw2)
print(yaw_diff)
nusc.render_sample_data(sd1['token']); plt.show()
nusc.render_sample_data(sd2['token']); plt.show()

It outputs:

90.04318656048041
135.87374587599848
-45.83055931551807  # Yaw diff on camera
-174.9605675915845
-128.6853167047343
-46.27525088685019  # Yaw diff on lidar

Note that the difference between -45.83 and -46.28 is likely due to lidar/camera sensors not being 100% aligned to the global axes. To get the accurate global yaw you should look at the raw data (before transforming it into the sensor frame).

from nuscenes-devkit.

@holger-motional thanks, i was using entire path instead of local path.

from nuscenes-devkit.

Hello,

Thank you very much for your reply. I implemented the function but I am still not sure about the yaw angle it returns.

As a case study I am considering 2 images:

n015-2018-08-01-16-32-59+0800__CAM_FRONT__1533112709162460
n015-2018-07-18-11-18-34+0800__CAM_FRONT__1531884168362460

(In the first image there is only one person, in the second I only look at the close person on the left side of the image with coordinates of the center as [-5.13221828 0.38146399 9.04940337])

For the first image I get an angle of 1.59 rad --> 91 degrees
For the second image (close person on the left of the image) 3.12 rad --> 179 degrees

Independently from the chosen convenction, The 2 people don't look as having 90 degrees difference between them. (More 45 degrees I'd say). Also looking at the rendered bounding boxes, the angle difference seems less than 90.

Am I missing something?

Thank you very much
Lorenzo

from nuscenes-devkit.

Great, thank you very much for your detailed explanation!

from nuscenes-devkit.

Hi, @holger-nutonomy , given the (yaw) orientation angle in global coordinate, how can we recover the corresponding quaternion? I found one line of code here: https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/utils/kitti.py#L315
This code simply sets the rotation axis to (0, 1, 0) in KITTI coordinate system. So is this the solution to recovering the quaternion? If so, then for the global nuscene coordinate system, should we similarly set the rotation axis to (0, 0, 1)?
Thank you.

from nuscenes-devkit.

@pxiangwu I suppose you are asking for nuScenes, not KITTI, right? Try this line:
Quaternion(axis=(0, 0, 1), angle=yaw)
https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/utils/tests/test_geometry_utils.py#L65

from nuscenes-devkit.

@pxiangwu I suppose you are asking for nuScenes, not KITTI, right? Try this line:
Quaternion(axis=(0, 0, 1), angle=yaw)
https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/utils/tests/test_geometry_utils.py#L65

Thank you so much. I think this is what I am looking for.

from nuscenes-devkit.

IndexError Traceback (most recent call last)
in ()
21 #nusc = NuScenes()
22
---> 23 sd1_im = [sd for sd in nusc.sample_data if sd['filename'] == '/content/data/set/nuscenes/samples/CAM_FRONT/n008-2018-08-01-15-16-36-0400__CAM_FRONT__1533151604012404.jpg'][0]
24 sd2_im = [sd for sd in nusc.sample_data if sd['filename'] == '/content/data/set/nuscenes/samples/CAM_FRONT/n008-2018-08-01-15-16-36-0400__CAM_FRONT__1533151605012404.jpg'][0]
25

IndexError: list index out of range

@holger-motional I am using the same code (#21 (comment)) for calculate yaw angles from camera, to crop field of view from lidar, and seems like i am missing some important point. any suggestions?

from nuscenes-devkit.

@programmeeer I don't think the sd['filename'] has your local path. It only stores the subpath, e.g. something like samples/CAM_FRONT/n008-2018-08-01-15-16-36-0400__CAM_FRONT__1533151605012404.jpg

from nuscenes-devkit.