A list of resources dedicated to compositionality.

Contributions most welcome! Please check the Contribution guideline for an example.

• Architectural Bias
• Datasets
• Interpretability
• Miscellaneous
  • Psycholinguistics
  • Cognitive Sciences

• David Ha, Andrew Dai, Quoc V. Le. HyperNetworks, arXiv:1609.09106, ICLR, 2017
  • The main idea is to use a hypernetwork to generate the weights for another network. An “embedding vector” is generated by a hyper RNN. This embedding vector is used to dynamically scale the weights of a RNN at every timestep. The approach is evaluated on different language tasks: character-level language modeling (PTB, Wiki), machine translation and handwriting prediction. The results outperform the baselines on these tasks marginally.

• Lake, Brenden, and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks ICML. 2018.

  • Study on the ability of RNNs to generalize in a compositional way on a data set for sequence-to-sequence modelling called SCAN, which requires models to create compositional solutions and generalize on new data based on related concepts.

• A Santoro, F Hill, D Barrett, A Morcos, T Lillicrap. Measuring abstract reasoning in neural networks ICML, 2018.

  • A dataset for abstract reasoning inspired by human-like IQ tests. To succeed in the task the model must be able to generalize in various ‘regimes’ in which the training and test data differ in clearlydefined ways.

• Atzmon, Yuval, et al. Learning to generalize to new compositions in image understanding arXiv:1608.07639 (2016).

  • A compositional split of the MS COCO dataset. This alternative split for training and test data can be used to test whether image captioning models can generalise to new (unseen) compositions.

• Singh, Chandan, et al. Hierarchical Interpretations for Neural Network Predictions arXiv:1806.05337 (2018).
  • Provides a method for creating a hierarchical representation of the contributions to the classification of both RNNs and CNNs.

• Christiansen, Morten H., and Nick Chater. The Now-or-Never bottleneck: A fundamental constraint on language. Behavioral and Brain Sciences 39 (2016).
  • Combining psycholinguistic evidence to hypothesize that the way humans process language is highly constrained by limited time and memory resources, therefore implying a very particular process in which the brain chunks incoming input and processes it into increasingly abstract representations.

• Lake, Brenden M., et al. Building machines that learn and think like people. Behavioral and Brain Sciences 40 (2017).
  • Highlighting the differences (and missing pieces) between human cognition and current Deep Learning architectures.