Input for training code2seq on new dataset about code2seq HOT 5 CLOSED

Hi @rishab-32 ,
Thank you for your interest in code2seq.

By default, our preprocessing takes Java method names as the code sequences. Getting the sequences from a separate text file will require some modifications. For example, you can change the method names to unique IDs in the Java file, and then use these IDs to look up the code sequence in the text file and join them.

Please see: https://github.com/tech-srl/code2seq#creating-and-preprocessing-a-new-java-dataset
and follow the execution of preprocess.sh on a small code directory, to see the outcomes of every step.

Best,
Uri

from code2seq.

Hi thanks for the prompt reply, In the paper, you have talked about an additional comparison to Hu et al. (2018) dataset. I am also working with the Hu dataset, do you have any implementation of code2seq available on this dataset? That would be really helpful.

from code2seq.

The comparison to Hu et al was rush, because one of the reviewers asked for it throughout the rebuttal. Since we found out that the BLEU scores reported by Hu et al were totally erroneous, we did not release official repro scripts (if I remember correctly, they reported exceptional BLEU score of about 40, while in fact, when I measured BLEU score on their logs - it was about 9).

However, I do have the scripts that I used to convert their dataset to our format.
If I remember correctly, you need to use the first script to convert their dataset from one-large-json into a directory of many files (for each of training/validation/test):

https://gist.github.com/urialon/6ce2ffab7b675d9437b730246dc07827

Then, run our preprocess.sh script on the resulting directories.
Finally, you need to "insert" the code sequences into the produced files, using this script:

https://gist.github.com/urialon/7c4d129f6cb45c2cdba669a15f132fb4

from code2seq.

But note that I haven't touched these scripts for almost two years, so it is very likely that I am forgetting some details. Follow these scripts carefully to verify that I did not mislead you.

from code2seq.

Thanks a lot for sharing the scripts. I understand that you do not have any intention of misleading me in any way.

from code2seq.