This repository is the implementation of our ECCV 2020 paper:

Active Visual Information Gathering for Vision-Language Navigation

Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, Jianbing Shen.

This work draws inspiration from human navigation behavior and endows an agent with an active information gathering ability for a more intelligent vision-language navigation policy.

To achieve this, we develop an active exploration module, which learns to 1) decide when the exploration is necessary, 2) identify which part of the surroundings is worth exploring, and 3) gather useful knowledge from the environment to support more robust navigation.

Please refer to our paper for the detailed formulations.

Here are some results on R2R dataset reported in our paper.

Please refer to our paper for the comparsions with previous arts.

1. Install Jupyter Install jupyter using the following scripts. pip install jupyter

2. Install R2R environment via Jupyter Our code is built basing on R2R-EnvDrop, please install the R2R environment for the python interpreter used in Jupyter following the installation instructions.

1. Download the checkpoint of the agent to directory snap/agent/state_dict/best_val_unseen. The checkpoint is available on Google Drive.
2. Start a Jupyter service and run the jupyter notebook evaluation.ipynb.

• Release the checkpoint.
• Add training code.

Please cite this paper in your publications if it helps your research:

@inproceedings{wang2020active,
    title={Active Visual Information Gathering for Vision-Language Navigation},
    author={Wang, Hanqing and Wang, Wenguan and Shu, Tianmin and Liang, Wei and Shen, Jianbing},
    booktitle=ECCV,
    year={2020}
}

Active VLN is freely available for non-commercial use, and may be redistributed under these conditions. Please see the license for further details. For commercial license, please contact the authors.

• hanqingwang[at]bit[dot]edu[dot]cn, Hanqing Wang
• wenguanwang[dot]ai[at]gmail[dot]com, Wenguan Wang