It was originally developed for use with the 50 Salads, GTEA, MERL Shopping, and JIGSAWS datasets. Recently we have also achieved high action segmentation performance on medical data, in robotics applications, and using accelerometer data from the UCI Smartphone dataset.
Numba: This makes the metrics much faster to compute but can be removed is necessary.
LCTM: Our older Conditional Random Field-based models.
Tested on Python 3.5. May work on Python 2.7 but is untested.
Contents (code folder)
TCN_main.py. -- Main script for evaluation. I suggest interactively working with this in an iPython shell.
compare_predictions.py -- Script to output stats on each set of predictions.
datasets.py -- Adapters for processing specific datasets with a common interface.
metrics.py -- Functions for computing other performance metrics. These usually take the form score(P, Y, bg_class) where P are the predictions, Y are the ground-truth labels, and bg_class is the background class.
tf_models.py -- Models built with TensorFlow / Keras.
utils.py -- Utilities for manipulating data.
Data
The features used for many of the datasets we use are linked below. The video features are the output of a Spatial CNN trained using image and motion information as mentioned in the paper. To get features from the MERL dataset talk to Bharat Signh at UMD.
Each set of features should be placed in the features folder (e.g., [TCN_directory]/features/GTEA/SpatialCNN/).
MERL Shopping: Email Bharat Signh at UMD for features.
Each .mat file contains three or four types of data: 'Y' refers to the ground truth action labels for each sequence, 'X' is the per-frame probability as output from a Spatial CNN applied to each frame of video, 'A' is the 128-dim intermediate fully connected layer from the Spatial CNN applied at each frame, and if available 'S' is the sensor data (accelerometer signals in 50 Salads, robot kinematics in JIGSAWS).
There are a set of corresponding splits for each dataset in [TCN_directory]/splits/[dataset]. These should be easy to use with the dataset loader included here.
I would like to thank you for sharing the code, it is a really great job. I would appreciate if you can tell me what are L and B values that you used for Dilated TCN. Also, I would like to know if you have the trained models.
Thank you!
id@ipadress:# python3 code/TCN_main.py
Using TensorFlow backend.
2020-06-22 02:48:46.974129: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
/workspace/code/utils.py:124: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "partition_latent_labels" failed type inference due to: Untyped global name 'segment_intervals': cannot determine Numba type of <class 'function'>
File "code/utils.py", line 130:
def partition_latent_labels(Yi, n_latent):
Zi = Yi.copy()
intervals = segment_intervals(Yi)
^
@jit("int64[:](int64[:], int64)")
/workspace/code/utils.py:124: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "partition_latent_labels" failed type inference due to: Untyped global name 'segment_intervals': cannot determine Numba type of <class 'function'>
File "code/utils.py", line 130:
def partition_latent_labels(Yi, n_latent):
Zi = Yi.copy()
intervals = segment_intervals(Yi)
^
@jit("int64[:](int64[:], int64)")
/usr/local/lib/python3.6/dist-packages/numba/core/object_mode_passes.py:178: NumbaWarning: Function "partition_latent_labels" was compiled in object mode without forceobj=True, but has lifted loops.
File "code/utils.py", line 126:
def partition_latent_labels(Yi, n_latent):
if n_latent == 1:
^
state.func_ir.loc))
/usr/local/lib/python3.6/dist-packages/numba/core/object_mode_passes.py:188: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
File "code/utils.py", line 126:
def partition_latent_labels(Yi, n_latent):
if n_latent == 1:
^
state.func_ir.loc))
/workspace/code/metrics.py:143: NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "levenstein_" failed type inference due to: No implementation of function Function() found for signature:
zeros(list(int64), Function(<class 'float'>))
There are 2 candidate implementations:
Of which 2 did not match due to:
Overload in function 'zeros': File: : Line <N/A>.
With argument(s): '(list(int64), Function(<class 'float'>))':
No match.
During: resolving callee type: Function()
During: typing of call at /workspace/code/metrics.py (147)
File "code/metrics.py", line 147:
def levenstein_(p,y, norm=False):
n_col = len(y)
D = np.zeros([m_row+1, n_col+1], np.float)
^
@jit("float64(int64[:], int64[:], boolean)")
/workspace/code/metrics.py:143: NumbaWarning:
Compilation is falling back to object mode WITHOUT looplifting enabled because Function "levenstein_" failed type inference due to: cannot determine Numba type of <class 'numba.core.dispatcher.LiftedLoop'>
File "code/metrics.py", line 148:
def levenstein_(p,y, norm=False):
D = np.zeros([m_row+1, n_col+1], np.float)
for i in range(m_row+1):
^
@jit("float64(int64[:], int64[:], boolean)")
/usr/local/lib/python3.6/dist-packages/numba/core/object_mode_passes.py:178: NumbaWarning: Function "levenstein_" was compiled in object mode without forceobj=True, but has lifted loops.
state.func_ir.loc))
/usr/local/lib/python3.6/dist-packages/numba/core/object_mode_passes.py:188: NumbaDeprecationWarning:
Fall-back from the nopython compilation path to the object mode compilation path has been detected, this is deprecated behaviour.
mid_video_SVM
/usr/local/lib/python3.6/dist-packages/numba/core/ir_utils.py:2031: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'p' of function 'levenstein_'.
File "code/metrics.py", line 153:
def levenstein_(p,y, norm=False):
for j in range(1, n_col+1):
^
warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
/usr/local/lib/python3.6/dist-packages/numba/core/ir_utils.py:2031: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'y' of function 'levenstein_'.
/usr/local/lib/python3.6/dist-packages/numpy/core/asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order, subok=True)
code/TCN_main.py:229: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
P_test = np.array(P_test)/float(n_classes-1)
code/TCN_main.py:230: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
y_test_ = np.array(y_test)/float(n_classes-1)
/usr/local/lib/python3.6/dist-packages/numpy/core/asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order, subok=True)
code/TCN_main.py:229: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
P_test = np.array(P_test)/float(n_classes-1)
code/TCN_main.py:230: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
y_test_ = np.array(y_test)/float(n_classes-1)
/usr/local/lib/python3.6/dist-packages/numpy/core/asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order, subok=True)
code/TCN_main.py:229: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
P_test = np.array(P_test)/float(n_classes-1)
code/TCN_main.py:230: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
y_test_ = np.array(y_test)/float(n_classes-1)
/usr/local/lib/python3.6/dist-packages/numpy/core/asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order, subok=True)
code/TCN_main.py:229: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
P_test = np.array(P_test)/float(n_classes-1)
code/TCN_main.py:230: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
y_test_ = np.array(y_test)/float(n_classes-1)
/usr/local/lib/python3.6/dist-packages/numpy/core/asarray.py:136: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
return array(a, dtype, copy=False, order=order, subok=True)
code/TCN_main.py:229: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
P_test = np.array(P_test)/float(n_classes-1)
code/TCN_main.py:230: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
y_test_ = np.array(y_test)/float(n_classes-1)
Hi Colin, thanks a lot for sharing this code, really helpful. Can I ask question about the implementation of you ED-TCN. Based on your paper(CVPR17), figure 1 shows there are two conv layers in the encoder and the decoder. However, your paper says "(e.g., 3 in the encoder)", and I can only see one conv layer in the encoder, "Convolution1D" line 81, tf_models.py. I might miss something there, I wonder which one is correct? Thanks.
Can you tell if the accelerometer features from 50 salads mid-level dataset(which I saw is a 30x1 vector at a timestep) are raw acceleration values or have they been processed?
Assuming the 30x1 vector has 3 subsequent values as acc_x,acc_y ,acc_z from each of the IMUs (which I believe were 8), are there 10 sensors or some values are expected to be empty?
I just want to make sure if I can use this for IMU data I have for action detection.
Can anybody tell me, why is there an error?
While I run the TCN_main.py, an error arises.
Traceback (most recent call last):
File "TCN_main.py", line 40, in
import tf_models, datasets, utils, metrics
File "/home/yangchihyuan/Research/TCN/code/metrics.py", line 8, in
from LCTM.IoU_metrics import *
ImportError: No module named 'LCTM.IoU_metrics'
The LCTM module was downloaded from the GitHub and I installed it using sudo -H python3 setup.py install
I run the file TCN_main.py using iPython and got an error message
/TCN/code/metrics.py in ()
3 from numba import jit, int64, boolean
4
----> 5 from LCTM import utils
6 import sklearn.metrics as sm
7
ImportError: No module named LCTM
I can't find the LCTM module in the utils.py file.
Why does it happen?
I appreciate any suggestion.
Hello, I would like to download the features of the 50 salada dataset. I want to train this model for academic research. I have submitted the request but have not received feedback. Can these trained features still be downloaded? Thank you for your reply.