cmu-perceptual-computing-lab / ssp Goto Github PK

View Code? Open in Web Editor NEW

30.0 6.0 9.0 86 KB

Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction (CVPR 2020)

Home Page: http://domedb.perception.cs.cmu.edu/ssp.html

Python 99.61% Shell 0.39%

ssp's Introduction

Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction (CVPR 2019 Oral)

Current version provides:

Haggling dataset from Panoptic Studio
Visualization code

Requirements

Python 3
PyOpenGL
freeglut (use sudo apt-get install freeglut3-dev for ubuntu users)

Download Processed Haggling Data

Run the following script to download processed haggling dataset (4.7GB)

sh ./scripts/download_haggling.sh

Each pkl file has information for each haggling game including 3D body, 3D face, 3D hand, speaking annotation for all three people
Speaking status annotation is in a binary form (0 or 1)

Training/Testing Split

The following sequences are testing set:

'170221_haggling_b1','170221_haggling_b2','170221_haggling_b3','170228_haggling_b1','170228_haggling_b2','170228_haggling_b3'

All others are training set
No subject appears in both training and testing set. They are completely seperate

Download Raw Data (with HD videos)

Use Panoptic-toolbox
In panoptic-toolbox, check script of script/getDB_haggling, and set the HD video numbers (between 1-31) you want to download

# $curPath/getData.sh 170221_haggling_b1 (vga_video_num)  (hd_video_num)
# For example, if you want to get all 3D videos, set 31 as follows 
$curPath/getData.sh 170221_haggling_b1 0 31

Run the following script in panoptic-toolbox

sh ./script/getDB_haggling.sh

Follow the Panoptic-DB instruction to extract videos to frames.
Note that these videos are raw videos, and you need to find corresponding frames for each haggling game. Check the start frame index of each haggling game (included in pkl files)

Visualization

python glViewer.py

mouseLeft: view rotation changes
mouseRight: zoom changes
shift+ mouseLeft: view pan
f: toggle floor drawing

Citation

@inproceedings{joo2019ssp,
  title={Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction},
  author={Joo, Hanbyul and Simon, Tomas and Cikara, Mina and Sheikh, Yaser},
  booktitle={CVPR},
  year={2019}
}

License

CC-BY-NC 4.0.

ssp's People

Contributors

Stargazers

Watchers

Forkers

yonghoonkwon heng-robotics xucong-zhang gazelei jpmh1309 isabella232 cualquiercosa327 bruinxiong

ssp's Issues

Where can I download the audio for the sequences?

Hello,

I have been trying to get my hands on the transcripts and the audio for the haggling sequences but seemingly they aren't available. On the panoptic project website I see that the example videos have audio but the download scripts aren't pulling the audio from anywhere.
Would you be kind enough to point me to the location where I can download the audio for the sequences @jhugestar?

is there any way i can get only one part like human hand ?

Subject startFrame and bValid in data

Hello!

I noticed that in addition to the startFrame in the pickled dictionary for each group, there is an entry for the start frame in the data dictionary for each subject as well. These don't match, so I am assuming that the start frame in the subject dictionary represents the first frame where they are visible in the scene. However, the data array for each subject has the same shape.

So I am wondering if the data is represented starting at the start frame for the haggling game, or the start frame mentioned in the data dictionary. Here is a running example using seq 170221_haggling_b1_group0:

In [23]: with open("170221_haggling_b1_group0.pkl", "rb") as f:
        ...:     data = pickle.load(f, encoding="latin1")

In [24]: data["startFrame"]
Out[24]: 1223

In [25]: data["subjects"][0].keys()
Out[25]: dict_keys(['joints19', 'bodyNormal', 'faceNormal', 'humanId', 'scores', 'startFrame', 'bValid'])

In [26]: data["subjects"][0]["startFrame"]
Out[26]: 1167

In [27]: data["subjects"][1]["startFrame"]
Out[27]: 1078

In [28]: data["subjects"][2]["startFrame"]
Out[28]: 1125

In [29]: data["subjects"][0]["bodyNormal"].shape
Out[29]: (3, 1832)

In [30]: data["subjects"][0]["joints19"].shape
Out[30]: (57, 1832)

In [31]: data["subjects"][1]["bodyNormal"].shape
Out[31]: (3, 1832)

In [32]: 1223 + 1832
Out[32]: 3055

Downloading the video for the sequence and extracting the individual frames, it seems like the participants start leaving shortly after frame 3055, leading me to believe that the data in the subjects dictionary is for the frames explicitly involving the game, and the startFrame in the subject dictionary is just auxiliary information not pertaining to the data.

On the same note of auxiliary data, how should one interpret bValid? It's a single boolean while the scores are provided for each frame, so I'm not sure how the two pieces combine.

Can someone confirm/clarify these please? Thanks!