BERT is the most powerful neural networks model in NLP area. Graph Neural Networks(GNN) is also one of the most popular model now. Therefore, we want to bring the advantages in BERT to the citation network, and evaluate that how much improvement can BERT help GNN models work.

In this repository, we will explore the use of fine-tuned BERT, how to encode node features using BERT model, and using citation graph and node features to train GNN models.

Now, it just support Pytorch framework!

The step is as follows:

1. Download ogbn-arxiv dataset from Open Graph Benchmark.
2. Fine-tuned BERT with arxiv dataset and select the model which has the best accuracy in validation set as encoder.
3. Encode the node features using fine-tuned BERT.
4. Train and evaluate the GNN model.

You can see the details about how to train the code in pytorch and tensorflow directory.

We've also created the following hands-on colab tutorials in pytorch and tensorflow that will allow you to get start faster:

• Pytorch

The experiments below are from this repo and Open Graph Benchmark Leaderboard.

All the results are the average at least over 10 runs.

For help or issues using our code, please contact Sung-Ping Chang ([email protected]).