This repository contains the code for the paper "Bridging Graph Position Encodings for Transformers with Weighted Graph-Walking Automata" (Soga and Chiang, 2023).
There are 2 main directories in this repository. graphs
contains code for running the graph dataset experiments, and mt
contains code for running the machine translation experiments.
To set up the environment for graphs
, we recommend creating a new virtual environment with Python 3.10. Run
python -m venv ~/.virtualenvs/gape-graphs
To install dependencies for graphs
, run the following:
cd graphs
source ~/.virtualenvs/gape-graphs/bin/activate
pip3 install -r requirements.txt
Dependencies for mt
require an environment with Python 3.7 instead, which is easy to install separate from your main installation with Conda. Run the following after deactivating gape-graphs
:
cd mt
conda env create -f environment.yml
conda activate gape-mt
Run the shell scripts in graphs/data/
to download the graph datasets. Run any preparation scripts in each dataset's respective folder. The only exception is OGB-PCQM4Mv2 which is downloaded automatically upon running graphs/scripts/run_OGB_graph_regression.sh
. The English-Vietnamese sentence pairs are already in mt/nmt/data
.
Bash scripts to test each PE per dataset are located in graphs/scripts
. By default, the scripts assume only one GPU and run experiments for each PE in series over 4 random seeds. The evaluation metrics and training outputs are logged in DEBUG.log
.
To verify the BLEU scores in the paper, run python bleu.py OUT-[model]-[expected BLEU] nmt/data/en2vi/test.vi.tok
substituting the model name and the expected BLEU score. To train a model, run train.sh
replacing the PE
variable in TASK=${SL}2${TL}_${PE}
with the desired PE scheme. See nmt/configurations.py
for the list of PE schemes and their hyperparameters.
Below is a table of all references and sources of code that is not ours.
Reference | Code | Repository |
---|---|---|
Dwivedi et al. (2020) | Graph transformer implementation, data preparation, and training & evaluation code | benchmarking-gnns |
Dwivedi et al. (2022) | Random walk PE implementation | gnn-lspe |
Kreuzer et al. (2021) | Spectral attention node PE implementation | SAN |
Nguyen | Transformer implementation for MT | witwicky |
Ying et al. (2021) | Shortest-path distance & centrality PE and Floyd-Warshall implementation | Graphormer |