Scene graph prediction is the task of mapping an image into a set of bounding boxes, along with their categories and relations (e.g., see [2, 3, 4, 5, 6]).

In the paper Scene Graphs with Permutation-Invariant Structured Prediction (2018) [1] we present a new architecture for graph inference that has the following structural property: on the one hand, the architecture is invariant to input permutations; on the other hand, every permutation-invariant function can be implemented via this architecture.

In this repository, we share our architecture implementation for the task of scene graph prediction.

Scene Graph Predictor (SGP) gets as an input inital confidience distributions per entity and relation and processes these to obtain new labels. SGP satisfies the graph permutation invariance property intoduced in the paper. The model is implemented in TensorFlow. For the initial confidence distributions per entity and relation, we simply re-use features learned by the baseline model from Zellers et al. (2017). (git repository https://github.com/rowanz/neural-motifs)

Our SGP implementation is using an iteratively RNN to process predictions. Each step outputs an improved predictions.

A schematic representation of the architecture. Given an image, a Label predictor outputs initial predictions . Then, our SGP model, computes each element wise. Next, they are summed to create vector , which is concatenated with . Then, is applied, and another summation creates the graph representation. Finally, classifies objects and classifies relation. The process of SGP could be repeated iteratively (in the paper we repeat it 3 times).

For more information, please look at the code (Module/Module.py file) and the paper.

Our SGP architecture uses attention at the feature-level for each node during inference. We weight the significance of each feature per node, such that the network can choose which features from adjacent nodes contributes the most information.

An example of attention per entities and global attention over all nodes. The size and location of objects provide a key signal to the attention mechanism. The model assigns higher confidence for the label "tie" when the label "shirt" is detected (third panel from the left). Similarly, the model assigns a higher confidence for the label "eye" when it is located near "hair".

To get started with the framework, install the following dependencies:

• Python 2.7
• tensorflow-gpu 1.0.1
• matplotlib 2.0.2
• h5py 2.7.0
• numpy 1.12.1
• pyyaml 3.12

Run "pip install -r requirements.txt" - to install all the requirements.

1. Run "python Run.py download" to download and extract train, validation and test data. The data already contains the result of applying the baseline detecor over the VisualGenome data.
2. Run "python Run.py eval gpi_linguistic_pretrained <gpu-number>" to evaluate the pre-trained model of our best variant, linguistic with multi-head attention. (recall@100 SG Classification).
3. Run "python Run.py train gpi_linguistic <gpu-number>" to train a new model (linguistic with multi-head attention).
4. Run "python Run.py eval gpi_linguistic_best <gpu-number>" to evaluate the new model. (recall@100 SG Classification).

This repository contains an implementation of our best variant (Linguistic with multi-head attention) of the Scene Graph Prediction (SGP) model introduced in the paper Scene Graphs with Permutation-Invariant Structured Prediction. (The repsitory updated for version 1 of the paper - the results of latest version will be published ×in the future). Specifically, the repository allow to run scene-graph classification (recall@100) evaluation script on our pre-trained model or alternatively (1) train an SGP model (2) evaluate the trained model using scene-graph classification (recall@100) evaluation script.

[1] Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, Amir Globerson, Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction, 2018.

[2] Justin Johnson, Ranjay Krishna, Michael Stark, Li Jia Li, David A. Shamma, Michael S. Bernstein, Fei Fei Li, Image Retrieval using Scene Graphs, CVPR, 2015.

[3] Cewu Lu, Ranjay Krishna, Michael S. Bernstein, Fei Fei Li, Visual Relationship Detection with Language Priors, ECCV, 2016.

[4] Xu, Danfei and Zhu, Yuke and Choy, Christopher and Fei-Fei, Li, Scene Graph Generation by Iterative Message Passing, CVPR, 2017.

[5] Alejandro Newell and Jia Deng, Pixels to Graphs by Associative Embedding, NIPS, 2017.

[6] Rowan Zellers, Mark Yatskar, Sam Thomson, Yejin Choi, Neural Motifs: Scene Graph Parsing with Global Context, CVPR, 2018.

Please cite our paper if you use this code in your own work:

@inproceedings{hr18perminvimg2sg,
  author    = {Roei Herzig and
               Moshiko Raboh and
               Gal Chechik and
               Jonathan Berant and
               Amir Globerson},
  title     = {Mapping Images to Scene Graphs with Permutation-Invariant Structured
               Prediction},
  booktitle = {Advances in Neural Information Processing Systems (NIPS)},
  year      = {2018}
}