tech-srl / code2vec Goto Github PK
View Code? Open in Web Editor NEWTensorFlow code for the neural network presented in the paper: "code2vec: Learning Distributed Representations of Code"
Home Page: https://code2vec.org
License: MIT License
TensorFlow code for the neural network presented in the paper: "code2vec: Learning Distributed Representations of Code"
Home Page: https://code2vec.org
License: MIT License
We are trying to fine-tune the code2vec model for a different task (a binary classification task).
We have previously tried to generate the codevectors for our code-snippets and feed them to a classifier (i.e. using code2vec as a feature extractor). The results haven't been what we expected so we were looking into fine-tuning code2vec with our data. We tried importing the meta_graph and wanted to retrieve the last tensor before classification in order to "connect" it to a binary output. Something along the lines of what is described in the section 'Using a pre-trained graph in a new graph' in https://blog.metaflow.fr/tensorflow-saving-restoring-and-mixing-multiple-models-c4c94d5d7125
Unfortunately we couldn't figure out the name of the tensor we required to be used in get_tensor_by_name(). We even opened the graph in Tensorboard to examine it but were a bit overwhelmed by the size of the graph.
Do you have any pointers as to how to modify the graph for a different classification task all while using the pre-trained weights from training code2vec with java-large?
Hi. Is the source code for the AST visualizer from the demo website available? I mean the script to which you feed the output of the /predict endpoint. It is very nice and I would like to use it for a university project if possible. Thank you!
Is it possible to provide this for other languages too, e.g. Python, JavaScript, etc.?
In FeatureExtractor, the length of a path is calculated as int pathLength = sourceStack.size() + targetStack.size() - 2 * commonPrefix;
(line 140). I believe this is calculation is one more than the actual path length as it is counting the commonNode twice.
I suggest calculating the length as currentSourceAncestorIndex + currentTargetAncestorIndex + 1
Hi, thanks for your fantastic work.
When I use code2vec to export vectors, and I try this code as Input.java:
public JdkCommunicatorLogger(Logger logger,Level logLevel,Level errorLogLevel){ if (logger == null) { throw new IllegalArgumentException("logger is required"); } if (logLevel == n ull) { throw new IllegalArgumentException("logLevel is required"); } if (errorLogLevel == null) { throw new IllegalArgumentException("exceptionLogLevel is required"); } this.logger=l o gger; this.logLevel=logLevel; this.errorLogLevel=errorLogLevel; }
I encountered this error:
Exception in thread "main" com.github.javaparser.ParseProblemException: Encountered unexpected token: "ull" <IDENTIFIER>
at line 1, column 215.
at com.github.javaparser.JavaParser.simplifiedParse(JavaParser.java:242)
at com.github.javaparser.JavaParser.parse(JavaParser.java:210)
at JavaExtractor.FeatureExtractor.parseFileWithRetries(FeatureExtractor.java:70)
at JavaExtractor.FeatureExtractor.extractFeatures(FeatureExtractor.java:40)
at JavaExtractor.ExtractFeaturesTask.extractSingleFile(ExtractFeaturesTask.java:64)
at JavaExtractor.ExtractFeaturesTask.processFile(ExtractFeaturesTask.java:39)
at JavaExtractor.App.main(App.java:33)
So, how can I avoid this error? Or can this be done?
On downloading the trained model, I get dictionaries.bin. However, on training the network with preprocessed data from https://s3.amazonaws.com/code2vec/data/java14m_data.tar.gz, I'm unable to generate dictionaries.bin at the end of the training.
Because of this, I am unable to use Interactive predict for manual examination of the model. Am I missing something here? Please let me know at which phase, dictionaries.bin would be generated.
Hi, I am trying to extend code2vec for Javascript. So far,I have been able to extract paths. I have a few questions about the final form of my_dataset.val.c2v.
What was the hash function used for paths? Did you use a standard hash function like sha1 or md5
Do you unhash the hashed string somewhere in the program?
Are the arrows in the path (up, down) really important?
How could I adapt code2vec to train the model for something else than the method's name?
In my case, I want the prediction to be a boolean.
When I try to run the training with tf2.0 in docker with gpu support, I get the following error:
docker run --runtime=nvidia -it -v $(realpath ~/Code):/code -u $(id -u):$(id -g) tensorflow/tensorflow:2.0.0a0-gpu-py3 bash
]Traceback (most recent call last):
File "code2vec.py", line 1, in <module>
from vocabularies import VocabType
File "/code/code2vec/vocabularies.py", line 45
self.word_to_index: Dict[str, int] = {}
^
SyntaxError: invalid syntax```
Can you guys provide a Dockerfile to start easily?
Hi, am I right that I can use ur NN for finding clones with similar semantics via vectors distance?
Thx for the answer.
Hello and thank you for your job!
I'm now working in code clones problem and I would like to use your approach for code representation. When I read your paper I wonder I can use your trained model for my problem, but I see nothing about that. How I can use your's model? As I see your output predictions contains only names for methods and attention scores, but I need just a vector of input java code. Is it possible?
Thank you!
Hi, I tried to run the model using your example. However if I run python3 code2vec.py --load models/java14_model/saved_model_iter8 --predict
I get the error:
Loading word frequencies dictionaries from: None ..
Where can i find .dict.c2v
file that is needed for self.config.word_freq_dict_path
?
PS I have already downloaded the weights of the network
Hi, @urialon
I have a question about the code of building model.
What is mask
doing?
https://github.com/tech-srl/code2vec/blob/master/model.py#L309-L311
I can predict that this code is related to updating parameters of attention weight
, but I could not read how it corresponds to the formula of the paper.
Thank you.
Hello,
I tried to play with Code2Vec, and specialized an already trained model with a my own dataset.
As suggested in #5, I used the following command line:
python3 -u code2vec.py --load ${trained_model} --data ${data} --test ${test_data} --save ${model_dir}/saved_model
However, I have the following errors:
Traceback (most recent call last):
File "code2vec.py", line 38, in <module>
model.train()
File "/tmp/code2vec/model.py", line 72, in train
self.config.NUM_EXAMPLES / self.config.BATCH_SIZE * self.config.SAVE_EVERY_EPOCHS), 1)
AttributeError: 'Config' object has no attribute 'NUM_EXAMPLES'
The field NUM_EXAMPLES
does not exist in Config object. I hacker model.py
and replace this value by 1
(I don't know what I'm doing, but just to see what happen! :-))
It seems that this field is also used in PathContextReader
Starting training
Traceback (most recent call last):
File "code2vec.py", line 38, in <module>
model.train()
File "/tmp/code2vec/model.py", line 76, in train
config=self.config)
File "/tmp/code2vec/PathContextReader.py", line 15, in __init__
self.batch_size = config.TEST_BATCH_SIZE if is_evaluating else min(config.BATCH_SIZE, config.NUM_EXAMPLES)
AttributeError: 'Config' object has no attribute 'NUM_EXAMPLES'
According to the code in common.py
, there is no such field in the Config
object:
def __init__(self):
self.NUM_EPOCHS = 0
self.SAVE_EVERY_EPOCHS = 0
self.BATCH_SIZE = 0
self.TEST_BATCH_SIZE = 0
self.READING_BATCH_SIZE = 0
self.NUM_BATCHING_THREADS = 0
self.BATCH_QUEUE_SIZE = 0
self.TRAIN_PATH = ''
self.TEST_PATH = ''
self.MAX_CONTEXTS = 0
self.WORDS_VOCAB_SIZE = 0
self.TARGET_VOCAB_SIZE = 0
self.PATHS_VOCAB_SIZE = 0
self.EMBEDDINGS_SIZE = 0
self.SAVE_PATH = ''
self.LOAD_PATH = ''
self.MAX_TO_KEEP = 0
self.RELEASE = False
self.EXPORT_CODE_VECTORS = False
What would be the default value of this field?
Hi~
When I used some examples in java-small dataset just like this,
@test public void f(){
assert MIDDLE_VERSIONS_ENTITY_NAME.equals(metadata().getEntityBinding(MIDDLE_VERSIONS_ENTITY_NAME).getTable().getName());
}
the website didn't show me the AST, I want to know whether the example must be out of the training dataset.
Thanks~
Hello! I've tried to train a model from scratch and received following error:
Average loss at batch 12300: 0.001828, throughput: 1302 samples/sec
Number of waiting examples in queue: 300000
2018-10-24 00:21:02.697593: E tensorflow/stream_executor/cuda/cuda_driver.cc:868] failed to alloc 4294967296 bytes on host: CUDA_ERROR_INVALID_VALUE: invalid argument
2018-10-24 00:21:02.787920: W ./tensorflow/core/common_runtime/gpu/cuda_host_allocator.h:44] could not allocate pinned host memory of size: 4294967296
It seems like something happened right at the end of first epoch.
I'm training a model in cloud with Tesla K80 and ~12 GB of RAM.
Do you have any idea on what could've caused an error?
Hi,
I am currently working on a research project regarding code similarity. For that, it would be really helpful for us if we could get the whole AST. From the code, I can get different number attention paths, but not the whole tree like the paper. Any guideline for that is really appreciated.
I am experimenting with code2vec for a binary classification problem for java methods, where the feature extractor generates the path vectors from AST as expected. The value of MAX_CONTEXTS in config.py is first set to 400. The preprocessing step works fine too with MAX_CONTEXTS set to 400 (in preprocess.sh) as the expected *.c2v files are generated.
Then when I start training (train.sh), I get the following error:
File "/home/lv71161/akarmakar/miniconda3/envs/c2v/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Expect 201 fields but have 401 in record 0
Could you tell what might be the issue here?
What am I doing wrong in the case where I set MAX_CONTEXTS = 400?
MAX_CONTEXTS = 200 works perfectly fine!
Hi, i am trying to use code2vec to generate representations(vectors) for some java code. I am using the following command:
python code2vec.py --load models/java14_model/saved_model_iter8.release --test --export_code_vectors
and a variation:
python code2vec.py --load models/java14_model/saved_model_iter8.release --test <file_with_filepaths> --export_code_vectors
in both cases i got the following errors
2019-11-02 23:46:57.585327: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
Traceback (most recent call last):
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Expect 201 fields but have 7 in record
[[{{node IteratorGetNext}}]]
[[IteratorGetNext/_19]]
(1) Invalid argument: Expect 201 fields but have 7 in record
[[{{node IteratorGetNext}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "code2vec.py", line 31, in
eval_results = model.evaluate()
File "/home/gustavoeloi/GU/UNICAMP/tese/code/code2vec/tensorflow_model.py", line 159, in evaluate
self.eval_original_names_op, self.eval_code_vectors],
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Expect 201 fields but have 7 in record
[[node IteratorGetNext (defined at /home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
[[IteratorGetNext/_19]]
(1) Invalid argument: Expect 201 fields but have 7 in record
[[node IteratorGetNext (defined at /home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1751) ]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'IteratorGetNext':
File "code2vec.py", line 31, in
eval_results = model.evaluate()
File "/home/gustavoeloi/GU/UNICAMP/tese/code/code2vec/tensorflow_model.py", line 122, in evaluate
input_tensors = input_iterator.get_next()
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
name=name)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2500, in iterator_get_next
output_shapes=output_shapes, name=name)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 793, in _apply_op_helper
op_def=op_def)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3360, in create_op
attrs, op_def, compute_device)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3429, in _create_op_internal
op_def=op_def)
File "/home/gustavoeloi/programs/anaconda3/envs/tf_gpu2/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1751, in init
self._traceback = tf_stack.extract_stack()
what should i do. Or, what am I doing wrong
Hello Urialon
I tried to run preprocess.sh for raw java, but It always gives me errors No such file or directory: 'java': 'java'
Could you check or give me solutions?
I really look forward to hearing from you
Regards
Dung
Hi Sir,
Your work is wonderful. The code is very readable and the instruction is very understandable. Thanks a lot.
Recently I'm applying your model to another data set and everything works well except --release and --export_code_vectors. Could you please give me a hand?
I run the code through Google Colab: python 3.6.8, tensorflow-gpu 2.0.0-beta1
The commands I used are as follows:
!python3 code2vec.py --framework keras --load models/test2vec/saved_model --release
!python3 code2vec.py --framework keras --load models/test2vec/saved_model --test data/my_dataset/my_dataset.test.c2v --export_code_vectors
The two commands run well and no errors, but I can not see any files generated, i.e. *.release and *.vectors can not be generated. I don't know why.
Could you please give me some hints? Thanks.
I have read there is already a keras implementation but as far as I understand this one is not compatibel with the weights of the pre trained models on S3. Is this correct? I'm wondering because me and my study group want to research if it is possible to use this network for transferlearning by replacing the final layer. However we find it easier to work with keras if possible.
Hello! I tried to extend the code2vec to Python, but I encountered some problems.
I successfully use PathMiner to create the preprocess data in a triple components format (Token, Path, Token), but I am not able to run bash train.sh because I still need dataname.dict.c2s.
What is dataname.dict.c2s? And how can I deal with it?
Hi,
I am trying to use code2vec for doing binary classification for a different language. So, instead of method names, i have True
and False
as the labels. My Custom feature extractor still generates the path vectors from AST in a similar manner. I have a relatively small training dataset.
The pre-process step works fine and generates *.c2v
files in data. However, while training, I am facing this error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [4128] vs. [6880] [[{{node LogicalAnd}}]]
The sizes of my training set is 4128, validation set is 644, test set is 641 . Do I need to modify something in common.py
for this?
Let me know if you need some other info or have better suggestions.
Thanks in advance.
In the code2vec paper published at POPL19, it says that the size of the code-vector is d with d=128. Moreover, also the vectors of the contexts after the dense layer should, according to the paper, have size 128. However, when debugging the code I noticed that their size is 3*d=384. Am I missing something?
Thank's for answering :)
Hi, I want to ask about the estraction of AST in your code.
Do you write the code manually or generate it with some framework?
I haven't fully understood your code, so i have to ask you here.
Thank you so much.
When I feed tool with a short method like this one.
@Override
public void andReturn(Object value) {
this.throwWrappedIllegalStateException();
}
I donot get the cod2vec, why?
Hi,
I have used the released model you have provided in the repository.
However, if I test it using the same codeblock in code2vec.org and locally, the values are different. I have just followed the instructions provided in the readme to run and predict.
Here's the codeblock to test:
void f(int[] list){
for(int i =1; i<list.length;i++){
int currentElement = list[i];
int k;
for(k =i-1;k>=0&&list[k]>currentElement;k--){
list[k+1]=list[k];
}
list[k+1]=currentElement;
}
}
The probability shown in code2vec site is insertionSort 82.35% , sortByInsertionSort
6.49%
The probability shown in the git project is (0.549103) predicted: ['sort', 'by', 'insertion', 'sort'], (0.095443) predicted: ['min', 'key'], (0.074358) predicted: ['insertion', 'sort']
Is the model used in the website and provided in the git repository different? Or I am missing something?
Thank you,
Zayed
In FeatureExtractor line 182, if (i == 0 || s_ParentTypeToAddChildId.contains(currentNode.getUserData(Common.PropertyKey).getRawType()))
the rawType of currentNode is checked. In the rest of the code (lines 156 and 167), the rawType of currentNode.getParentNode() is checked. Is this an intentional difference? If so why?
Hi,
I would like to use the pre-trained model to find the most similar name of a given word, as similar to the "Most similar feature" in the website https://code2vec.org .
However, when I input a method name that is out of the method name vocabulary (I guest that), for example "bitcounts", I got the error "word 'bitcounts' not in vocabulary".
Could you show me how to over-come this issue.
My code is as bellow:
from gensim.models import KeyedVectors as word2vec
vectors_text_path = 'C:/workplace/code2vec/models/java14_model/targets.txt'
model = word2vec.load_word2vec_format(vectors_text_path, binary=False)
print("input method name: ")
method_name = input()
print("most similar: ")
rs = model.wv.similar_by_word(method_name)
print(rs)
Thank you in advance
Hello I really wish to adapt your code to C/C++
Do you have any idea to do this, I really appreciate you help
I look forward to hearing from you
This is what I would like to do:
Given a folder with source code files, generate a file with the vectors that represent each method, but instead of having only the name of the methods I would like some additional data to know the location of the methods (file name, start line, start column, end line, end column)
Is there any way I can achieve this output?
Thanks for the impressive work.
I want to use the word2vec to transfer learning for some other downstream tasks: I have got some java code snippets and I want to get the representation of these methods. I found the parameter export_code_vectors but it seems that it is not implemented. Is that deleted or can you give me some advice on how to use it?
Hi @urialon,
I'm looking at your preprocessor.sh file but I don't know how this histo.tgt.c2v file is generated so I have two questions:
1/ How this file being generated?
2/ What this file does?
Thanks.
Hi, thank you for providing the official implementation of you outstanding paper.
I found that you recently removed the paper-version
branch from the repo. 1c92dab
I'd like to investigate your original implementation in detail in comparison with the current master
branch.
I would be grateful if you could consider to restore the branch.
Thank you.
Hi! I tried to run code2vec model and faced with problem associated with mistake in README
I used trained model from https://s3.amazonaws.com/code2vec/model/java14m_model.tar.gz
When I run:
python3 code2vec.py --load models/java14m/saved_model_iter8 --predict
I saw this
Unsuccessful TensorSliceReader constructor: Failed to find any matching files for models/java14m/saved_model_iter8
The reason is that archive contains model with name "saved_model_iter8.release"
If we run like below all will be fine
python3 code2vec.py --load models/java14m/saved_model_iter8.release --predict
Please fix README or rename model in archive. Thanks
Hello, I'm interested in this project.
But when I try to evaluate your network through your java14m dataset, there is a small error in the Dataset.map function in the self._create_dataset_pipeline. The error is "TypeError: Tensor is unhashable if Tensor equality is enabled. Instead, use tensor.experimental_ref() as the key.".
Coud you give me some help for this question? Thank you.
Hello,
I'm trying to run the command
python3 code2vec.py --load models/java14_model/saved_model_iter8.release --predict
but i found these error below.:
~/code2vec$ python3 code2vec.py --load models/java14_model/saved_model_iter8.release --predict
2019-11-13 00:48:00.400601: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:48:00.400716: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2019-11-13 00:50:06.233454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-11-13 00:50:06.808643: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 00:50:06.813225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 950M computeCapability: 5.0
coreClock: 0.928GHz coreCount: 5 deviceMemorySize: 3.95GiB deviceMemoryBandwidth: 74.65GiB/s
2019-11-13 00:50:06.813965: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:50:06.814661: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:50:06.815327: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:50:06.816016: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:50:06.816684: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:50:06.817445: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2019-11-13 00:50:11.048508: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-11-13 00:50:11.048636: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1592] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-11-13 00:50:11.050286: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-13 00:50:11.541359: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2712000000 Hz
2019-11-13 00:50:11.542624: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55cd55e30800 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-11-13 00:50:11.542733: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2019-11-13 00:50:12.295303: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-13 00:50:12.297222: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55cd56f8ec30 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2019-11-13 00:50:12.297333: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 950M, Compute Capability 5.0
2019-11-13 00:50:12.297654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-13 00:50:12.297710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]
2019-11-13 00:50:12,299 INFO
2019-11-13 00:50:12,300 INFO
2019-11-13 00:50:12,300 INFO ---------------------------------------------------------------------
2019-11-13 00:50:12,300 INFO ---------------------------------------------------------------------
2019-11-13 00:50:12,301 INFO ---------------------- Creating word2vec model ----------------------
2019-11-13 00:50:12,301 INFO ---------------------------------------------------------------------
2019-11-13 00:50:12,301 INFO ---------------------------------------------------------------------
2019-11-13 00:50:12,301 INFO Checking number of examples ...
2019-11-13 00:50:12,302 INFO ---------------------------------------------------------------------
2019-11-13 00:50:12,302 INFO ----------------- Configuration - Hyper Parameters ------------------
2019-11-13 00:50:12,303 INFO CODE_VECTOR_SIZE 384
2019-11-13 00:50:12,303 INFO CSV_BUFFER_SIZE 104857600
2019-11-13 00:50:12,304 INFO DEFAULT_EMBEDDINGS_SIZE 128
2019-11-13 00:50:12,304 INFO DL_FRAMEWORK tensorflow
2019-11-13 00:50:12,304 INFO DROPOUT_KEEP_RATE 0.75
2019-11-13 00:50:12,305 INFO EXPORT_CODE_VECTORS False
2019-11-13 00:50:12,305 INFO LOGS_PATH None
2019-11-13 00:50:12,305 INFO MAX_CONTEXTS 200
2019-11-13 00:50:12,306 INFO MAX_PATH_VOCAB_SIZE 911417
2019-11-13 00:50:12,306 INFO MAX_TARGET_VOCAB_SIZE 261245
2019-11-13 00:50:12,306 INFO MAX_TOKEN_VOCAB_SIZE 1301136
2019-11-13 00:50:12,306 INFO MAX_TO_KEEP 10
2019-11-13 00:50:12,306 INFO MODEL_LOAD_PATH models/java14_model/saved_model_iter8.release
2019-11-13 00:50:12,307 INFO MODEL_SAVE_PATH None
2019-11-13 00:50:12,307 INFO NUM_BATCHES_TO_LOG_PROGRESS 100
2019-11-13 00:50:12,307 INFO NUM_TEST_EXAMPLES 0
2019-11-13 00:50:12,307 INFO NUM_TRAIN_BATCHES_TO_EVALUATE 1800
2019-11-13 00:50:12,307 INFO NUM_TRAIN_EPOCHS 20
2019-11-13 00:50:12,308 INFO NUM_TRAIN_EXAMPLES 0
2019-11-13 00:50:12,308 INFO PATH_EMBEDDINGS_SIZE 128
2019-11-13 00:50:12,308 INFO PREDICT True
2019-11-13 00:50:12,308 INFO READER_NUM_PARALLEL_BATCHES 6
2019-11-13 00:50:12,308 INFO RELEASE False
2019-11-13 00:50:12,309 INFO SAVE_EVERY_EPOCHS 1
2019-11-13 00:50:12,309 INFO SAVE_T2V None
2019-11-13 00:50:12,309 INFO SAVE_W2V None
2019-11-13 00:50:12,309 INFO SEPARATE_OOV_AND_PAD False
2019-11-13 00:50:12,309 INFO SHUFFLE_BUFFER_SIZE 10000
2019-11-13 00:50:12,309 INFO TARGET_EMBEDDINGS_SIZE 384
2019-11-13 00:50:12,310 INFO TEST_BATCH_SIZE 1024
2019-11-13 00:50:12,310 INFO TEST_DATA_PATH None
2019-11-13 00:50:12,310 INFO TOKEN_EMBEDDINGS_SIZE 128
2019-11-13 00:50:12,310 INFO TOP_K_WORDS_CONSIDERED_DURING_PREDICTION 10
2019-11-13 00:50:12,310 INFO TRAIN_BATCH_SIZE 1024
2019-11-13 00:50:12,311 INFO TRAIN_DATA_PATH_PREFIX None
2019-11-13 00:50:12,311 INFO USE_TENSORBOARD False
2019-11-13 00:50:12,311 INFO VERBOSE_MODE 1
2019-11-13 00:50:12,311 INFO _Config__logger <Logger code2vec (INFO)>
2019-11-13 00:50:12,311 INFO context_vector_size 384
2019-11-13 00:50:12,312 INFO entire_model_load_path models/java14_model/saved_model_iter8.release__entire-model
2019-11-13 00:50:12,312 INFO entire_model_save_path None
2019-11-13 00:50:12,312 INFO is_loading True
2019-11-13 00:50:12,312 INFO is_saving False
2019-11-13 00:50:12,312 INFO is_testing False
2019-11-13 00:50:12,313 INFO is_training False
2019-11-13 00:50:12,313 INFO model_load_dir models/java14_model
2019-11-13 00:50:12,313 INFO model_weights_load_path models/java14_model/saved_model_iter8.release__only-weights
2019-11-13 00:50:12,313 INFO model_weights_save_path None
2019-11-13 00:50:12,313 INFO test_steps 0
2019-11-13 00:50:12,313 INFO train_data_path None
2019-11-13 00:50:12,313 INFO train_steps_per_epoch 0
2019-11-13 00:50:12,313 INFO word_freq_dict_path None
2019-11-13 00:50:12,313 INFO ---------------------------------------------------------------------
2019-11-13 00:50:12,314 INFO Loading model vocabularies from: models/java14_model/dictionaries.bin
...
2019-11-13 00:50:28,240 INFO Done loading model vocabularies.
2019-11-13 00:50:30,001 INFO Done creating code2vec model
2019-11-13 00:50:38.261557: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-13 00:50:38.261595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]
WARNING:tensorflow:From /home/sidekoiii/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Traceback (most recent call last):
File "code2vec.py", line 36, in
predictor = InteractivePredictor(config, model)
File "/home/sidekoiii/code2vec/interactive_predict.py", line 16, in init
model.predict([])
File "/home/sidekoiii/code2vec/tensorflow_model.py", line 321, in predict
self._build_tf_test_graph(reader_output, normalize_scores=True)
File "/home/sidekoiii/code2vec/tensorflow_model.py", line 287, in _build_tf_test_graph
input_tensors = _TFEvaluateModelInputTensorsFormer().from_model_input_form(input_tensors)
File "/home/sidekoiii/code2vec/tensorflow_model.py", line 528, in from_model_input_form
path_source_token_strings=input_row[5],
IndexError: tuple index out of range
However I can run this command successfully
python3 code2vec.py --load models/java14_model/saved_model_iter8.release --test data/java14m/java14m.test.c2v
Hi Uri,
I'm confused about code vecor and target embedding. In Readme:
Exporting the trained target (method name) embeddings:
python3 code2vec.py --load models/java14_model/saved_model_iter8 --save_t2v models/java14_model/targets.txt
Exporting the code vectors for the given code examples
The flag --export_code_vectors allows to export the code vectors for the given examples.
HI, Thanks for your contributions.
I'm concerned with shadowing code2vec, and wishing to do it from the scratch.(including preprocess raw data to preprocessed dataset)
So, I wanna ask you that offers raw dataset or not!
Is the only way to get raw dataset is crawling myself? ๐ข
I look forward to hearing from you soon.
Sincerely,
Hyuna Shin
Hi,
First of all, I would like to thank you for this good job, it is awesome!
Then I would like to ask two questions:
I want to train a model of the name convention of the test suite. What should I do to train the model? Should make the scripts pointings to the root of the test suite, e.g. src/test/java/
in a typical maven project? Or Should I extract each test method from their test classes and create a large fake java file with all (and only) the test methods, such as the Input.java file when we predict the label of a method?
What would you suggest in the case that we do not have a lot of data to train the model?
Thank you again.
Hi, when I begin to checked the dataset "java14m.dict.c2v", I found the format like this
"[path, -1354628, 30]". "Path" and "30" are "Xs" and "Xt", and "-1354628" is the path. And the number is node.
I have a question about that:
why the path < 0 ? because of "โ" "โ" ? So could you tell me how to represent it?
I'm confused about that, although the Definition2 and Defination3 have mentioned it.
Thank you~
I am so sorry to bother you again.
I want to know where I can find the raw data and I checked the issues list, but I cannot find it on http://urialon.cswp.cs.technion.ac.il/publications/. And I just see [PDF][Slides] [Video][Code and trained model][BibTeX] in the block of code2vector.
Best regards,
Wang Kun
Thanks for opening source this, it's great work.
I have been trying to run it on another dataset, specifically the java-small
dataset from your code2seq work which I found at https://urialon.cswp.cs.technion.ac.il/publications/.
The issue is that the preprocess.sh
script seems to get "stuck" extracting paths from the training set.
I say "stuck" because I'm not actually sure if the script has frozen or it just seems to take a long time (has currently been running for >3 hours).
Do you have an ETA on how long it took from your code2seq experiments? Or will these scripts not work with those datasets?
Thanks in advance.
ValueError: The passed save_path is not a valid checkpoint: models/java14_model/saved_model_iter8
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.