Hi, I'm looking into this repo and wondering small details such as,

• format of training data
• what problems it can solve
• how to use it after training and testing
• How to generate embeddings?

Is there any documentation / article hosted?

The link provided in README does not work any more.

Hello, I have some questions when running your code:
For PPI data, I downloaded your ppi.zip, which has .npy file. when Runing code，
ValueError: File suffix must be .json, .json.gz, .pkl or .pkl.gz: data / ppi \ train_feats.npy
How can I fix it ?

Looking at your QM9 dataset I've discovered that the raw data seems to be standardized. Here you seem to provide the targets' standard deviations. Comparing (the inverse of) these with the standard deviations (and converting all energies from kcal to eV) I mostly get a good match with the PyTorch Geometric dataset's standard deviations:

 Target: GNN-FiLM ?= PyG
      μ:    15.03 ?= 1.50
      α:    81.73 ?= 8.18
 ε_homo:     0.60 ?= 0.60
 ε_lumo:     1.29 ?= 1.27
     Δε:     1.29 ?= 1.29
   <R²>:   233.73 ?= 280.61
   ZPVE:    32.58 ?= 0.90
U0_atom:    10.41 ?= 10.33
 U_atom:    10.50 ?= 10.42
 H_atom:    10.58 ?= 10.50
 G_atom:     9.58 ?= 9.51
    c_v:    81.35 ?= 4.06

However, there is a 10x difference in μ and α and <R²> (gap), ZPVE, and c_v don't match at all. What am I missing here?

PS.: There seems to be a misunderstanding of the term chemical accuracy. The chemical accuracy would be a constant associated with the precision of experimental measurements, not the standard deviation . In issue #11 you said that CHEMICAL_ACC_NORMALISING_FACTORS gives the chemical accuracies. Those (and neither their inverse or some other transformation) don't match at all:

 Target: GNN-FiLM ?= MPNN chem. accuracy
      μ:   0.0665 ?= 0.100
      α:   0.0122 ?= 0.100
 ε_homo:   0.0719 ?= 0.043
 ε_lumo:   0.0337 ?= 0.043
     Δε:   0.0335 ?= 0.043
   <R²>:   0.0043 ?= 1.200
   ZPVE:   0.0013 ?= 0.001
     U0:   0.0042 ?= 0.043
      U:   0.0041 ?= 0.043
      H:   0.0041 ?= 0.043
      G:   0.0045 ?= 0.043
    c_v:   0.0123 ?= 0.050
      ω:   0.0375 ?= 10.000

Hi, I am trying to understand the "VarMisuse" task and i have run the code successfully. While I am reading dataset of the task, I find the dataset contains pre-generated graphs stored in *.gz from source projects. However, I cannot find code about the graph generation. If it is convenient, could you please tell me is there any code about the preprocessing?
Thanks a lot!

Hello, I found a performance issue in the definition of sparse_rgdcn_layer, gnns/rgdcn.py, tf.initializers.truncated_normal will created repeatedly during program execution, resulting in reduced efficiency. I think it should be created before the loop in sparse_rgdcn_layer.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

Hey @mmjb, how are you?

I am trying to play with training/testing of different architectures for VarMisuse, and I am trying to reproduce the exact dataset as in your experiments from the GNN-FiLM paper, such that the numbers will be comparable.

I downloaded and decompressed the dataset into data/varmisuse/.
The code assumes that there is graphs-test subdirectory and a graphs-testonly directory, but in fact, the dataset is split to projects first, and then to train/dev/test.

I guess that I can re-split according to Appendix D in your ICLR'2018 paper, but there are a few inconsistencies between the Appendix and the actual dataset, for example: openlivewriter appears in the dataset but not in the paper, it only has a graphs subdir but not graphs-{train,test,valid}; hangfire, optikey (train) and ravendb(dev) are in the paper but not in the dataset; botbuilder is marked as "train" in the paper but its graphs-train subdir is empty (unlike the rest of the training projects).

So -

1. Did you use the paper version of the dataset or the public version for the experiments reported in the GNN-FiLM paper?
2. If the answer is "public" - then I should just re-organize the files to create the training, test, and test-only dirs? In this case, the "dev" set contains only a single project, right?
3. Regarding file types - when I try to load a trained model and test it on a directory which contains only the file commandline.0.gz - I'm getting an error in richpath.py that "ValueError: File suffix must be .json, .json.gz, .pkl or .pkl.gz: data/varmisuse/commandline/graphs/commandline.0.gz".

Am I missing something? Maybe it will be easiest if you could share your file structure in the data/varmisuse dir and I'll organize my files accordingly?

Thanks a lot!

Thank you for making the code public and helping with the target value scaling last time!

Looking at the source code I've now discovered that you are using the QM9 atom features as-is. However, your version of the dataset includes the Mulliken partial charges, as described in the original paper: https://www.nature.com/articles/sdata201422/

It is rather easy to spot since it is the only float-valued node feature (number 7). This feature should not be available to the model. You can only obtain these charges via QM-based calculations based on the molecular geometries. This is therefore an output, not an input.

In my opinion, this feature should be excluded from the dataset, as done e.g. in the Gilmer MPNN paper. Otherwise you have information leakage. This seems to be an accident -- or did I overlook something? What are your thoughts?

when I run "python train.py RGCN PPI", there is an error "No such file or directory: 'data/ppi/train_graph.json'
"

I would like to get Precision and Recall metrics during the training of the Varmisuse task. I have tried modifying varmisuse_task.py with the following code:

Inside make_task_output_model, at approximately line 438
`predicted = tf.argmax(tf.nn.softmax(logits), 1, output_type=tf.int32)
prediction_is_correct = tf.equal(predicted, correct_choices)
accuracy = tf.reduce_mean(tf.cast(prediction_is_correct, tf.float32))

    TP = tf.count_nonzero(predicted * correct_choices)
    TN = tf.count_nonzero((1-predicted) * (1-correct_choices))
    FP = tf.count_nonzero(predicted * (1-correct_choices))
    FN = tf.count_nonzero((1-predicted) * correct_choices)

    precision = tf.divide(TP, TP+FP)
    recall = tf.divide(TP, TP+FN)

    tf.summary.scalar('accuracy', accuracy)
    model_ops['task_metrics'] = {
        'loss': tf.reduce_mean(per_graph_loss),
        'total_loss': tf.reduce_sum(per_graph_loss),
        'accuracy': accuracy,
        'precision': precision,
        'recall': recall,
        'num_correct_predictions': tf.reduce_sum(tf.cast(prediction_is_correct, tf.int32)),
    }`

Inside pretty_print_epoch_task_metrics:

acc = sum([m['num_correct_predictions'] for m in task_metric_results]) / float(num_graphs) precision = sum([m['precision'] for m in task_metric_results]) / float(num_graphs) recall = sum([m['recall'] for m in task_metric_results]) / float(num_graphs) return "Accuracy: %.3f | Precision: %.3f | Recall: %.3f" % (acc, precision, recall)

However, this code outputs nan for both Precision and Recall. If anyone knows why this happens and could point me in the right direction, I would greatly appreciate it.

This is a great project! I'd like to run experiments on the VarMisuse task, and ideally on other related tasks like variable naming. The training process works fine for me, but how can I access the raw C# programs used to create the dataset? Alternatively, is there code for creating the graphs from source C# programs? I'd like to construct different graph structures, e.g. by preprocessing the programs, performing program analyses etc.

I tried to reconstruct the programs using the 'ContextGraph' property on samples in the dataset, but the programs don't seem to be correct (e.g. only 2 blocks are closed for an Akka program).

In [43]: program = ""
    ...: for u, v in raw_sample['ContextGraph']['Edges']['NextToken']:
    ...:     label = raw_sample['ContextGraph']['NodeLabels'][str(u)]
    ...:     program += label
    ...:     if label == ";":
    ...:         program += "\n"
    ...: program += raw_sample['ContextGraph']['NodeLabels'][str(v)]
    ...: print(program)

(,value)<SLOT>valuevaluevaluekvkvkvkv_clusterbucketvbucketkeyv_cluster_cluster_cluster_cluster_clusterbucketbucketbucketvkeykeyContext_cluster_cluster_cluster_cluster_cluster_cluster_cluster_cluster_cluster_clusterbucketValueHolder,v,=vvar,)vvdeltaContent=bucketkvkvcurrentkvcurrentkvkv>&&=entryvar.entryentry.entryentryentry.=>entry(.)(=>kv.Count;
Sender.SenderTell(count)countvar=_registry._registry_registry_registrySum{=bucketbucketvar{)foreach(varentryin_registrykvin.{_registryvartopicPrefix=Self.SelfSelf=Key;
...

Thanks!

Hello

I have tried read the varmisuse implementation provided by varmisuse_task.py, however I faced one question on building the subtoken nodes, see the following line

Thanks

Hi, when I try to unzip the varmisuse dataset, it says

$7za x ../graph-dataset.zip

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,32 CPUs AMD Ryzen Threadripper 2950X 16-Core Processor  (800F82),ASM,AES-NI)

Scanning the drive for archives:
1 file, 14654161628 bytes (14 GiB)

Extracting archive: ../graph-dataset.zip

ERRORS:
Headers Error

--
Path = ../graph-dataset.zip
Type = zip
ERRORS:
Headers Error
Physical Size = 14654161628
64-bit = +



Archives with Errors: 1

Open Errors: 1

However, there are still some files get extracted. I wonder if this error is expected?

Sorry to bother you.
I want to classify graphs (each graph has a label and contains multiple nodes). I have read the cases in your project. If I want to fulfill my needs,
pm9 is a regression problem. I want to change it to a classification problem.
ppi is the classification of the nodes, and I want to change to the classification of the graph.
What is the fastest way to do it, and I hope to get your suggestions,thank u.

Hi,Good evening~
I am confused about using reorg_varmisuse_data file and I need your help.
I know that when this file runs to the end, the varmisuse_data_splitter.py file is not executed,
The error is as follows：
“
File "./utils/varmisuse_data_splitter.py", line 26
def _data_loading_worker(file_queue:Queue, result_queue:Queue) -> None:
^
SyntaxError: invalid syntax
”
I tried to solve this problem, but it didn't work.If you can provide some help, thank you so much!!!!

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

While training the VarMisuse task with GGNN model, I have noticed that results are not reproducible even with the random seed fixed.

What I have done:

1. Fixed the random seed in VarMisuse_GGNN.json to any positive value (e.g., 1252).
2. Trained two models using python train.py GGNN varmisuse --data-path <my_data_path> --result-dir <my_result_dir>
   What I expect:
   With the same random seed, the results of training on the same dataset should be the exact same, reflected in the log files generated in the result dir.

What I get instead:
Different results, reflected in different training and validation accuracies and different numbers of epochs trained.

I was wondering whether this is an implementation error, or if there is a reason that fixing the random seed does not make results reproducible that I am overlooking.

Thank you!

I find the target values in /data/qm9/train.jsonl.gz do not match to its real value in the original dataset.

For example, the QM9 id 0000001 molecule should be methane, therefore the first value of dipole should be 0 instead of [-1.7779076]. Besides, all other 11 targets lose its physical meanings.

{"targets": [[-1.7779076], [-7.5946741], [-6.7142577], [2.2468657], [5.355917], [-4.114645], [-3.1489365], [5.7098937], [5.6933656], [5.6850829], [5.7576447], [-6.1835322], [-1.3203824]], "graph": [[0, 1, 1], [0, 1, 2], [0, 1, 3], [0, 1, 4]], "id": "qm9:000001", "node_features": [[0, 1, 0, 0, 0, 6, -0.535689, 0, 0, 0, 0, 0, 1, 0, 4], [1, 0, 0, 0, 0, 1, 0.133921, 0, 0, 0, 0, 0, 0, 1, 0], [1, 0, 0, 0, 0, 1, 0.133922, 0, 0, 0, 0, 0, 0, 1, 0], [1, 0, 0, 0, 0, 1, 0.13392299, 0, 0, 0, 0, 0, 0, 1, 0], [1, 0, 0, 0, 0, 1, 0.13392299,0, 0, 0, 0, 0, 0, 1, 0]]}

The orginal value of QM9:000001 molecule methane should be as follow:

smiles | mu | alpha | homo | lumo | gap | r2 | zpve | cv | u0 | u298 | h298 | g298
C | 0 | 13.21 | -0.3877 | 0.1171 | 0.5048 | 35.3641 | 0.044749 | 6.469 | -40.4789 | -40.4761 | -40.4751 | -40.4986

What method did you use for QM9 target preprocessing? How can we scale back to the MAE in Chemical units according the paper? Glad if you can help. The units for reference may be shown in [https://arxiv.org/pdf/1712.06113v3.pdf].

Hi,

Thanks for the code.
Different papers seem to report different units and there is large difference in the errors of these papers like MPNN and k-gnn ( https://arxiv.org/pdf/1810.02244.pdf ). Can you let me know what are the units of targets for QM9 dataset reported in the paper?

when use params "no_parallel=True" in _load_data() ,We can get the data successfully from load_single_sample method , and reurn result list[GraphSample].but __load_data() can't get the result from “return _load_data()” ,or the type of the result that should be a list is a generator.
the code as follow:

the exception:

the debug info in vscode:


In order to facilitate debugging, we will assign the result to the temporary variable 'result' first.It is easy to find that there is correct data in ‘result’, but there is no data in self.loaded_data

Hi!
I am just a fan of state of the art, not an NLP researcher.

I found on NLP-progress this SOTA result on dependency parsing using GNN.
https://www.aclweb.org/anthology/P19-1237
Source: https://github.com/sebastianruder/NLP-progress/blob/master/english/dependency_parsing.md

I believe they use a GAT, and they claim to be the first to use a GNN for this fundamental NLP task.

I was thinking that, you made what seems to be like a major, task generic advance in GNNs, and that it would be interesting for you to make a collaboration with those researchers.

I am trying run these examples but was unable to set up a proper environment
Is there a guide or is this code abandoned?
It would be helpful to have a docker