dru-mara / evalne Goto Github PK
View Code? Open in Web Editor NEWSource code for EvalNE, a Python library for evaluating Network Embedding methods.
Home Page: https://evalne.readthedocs.io/en/latest/
License: MIT License
Source code for EvalNE, a Python library for evaluating Network Embedding methods.
Home Page: https://evalne.readthedocs.io/en/latest/
License: MIT License
Hi, I found out when performing a benchmark experiment that at a certain point all available threads were used, this killed the process of a colleague of mine. I found the culprit to be the construction of the edge embeddings, this is written efficiently with Numpy but by default uses all available threads.
I tried to alter this behavior via joblib backend but that did not work.
I found solution which was to include the following 2 lines before importing numpy, in the script where I call the evalne evaluator:
import os
os.environ["OPENBLAS_NUM_THREADS"] = "24"
Note that the number of threads need to be given as a str not an int.
Hope this helps others with the same problem.
Hello, I'm testing tadw of openne library with simple-example.py file. There are the following errors. I hope you can help me solve the questions in your busy schedule.
D:\software\Python3.5\python3.exe "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydevd.py" --multiproc --qt-support=auto --client 127.0.0.1 --port 5345 --file C:/Users/liujinxin/Desktop/EvalNE-master/examples/simple-example.py
pydev debugger: process 1788 is connecting
Connected to pydev debugger (build 183.5912.18)
Running command...
python3 -m openne --method tadw --input C:/Users/liujinxin/Desktop/xiugai/OpenNE-master/dwata/cora/cora_edgelist.txt --graph-format edgelist --output vec_all.txt --q 0.25 --p 0.25 --input ./edgelist.tmp --output ./emb.tmp --representation-size 128
Reading...
Traceback (most recent call last):
File "D:\software\Python3.5\lib\runpy.py", line 193, in run_module_as_main
"main", mod_spec)
File "D:\software\Python3.5\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "D:\software\Python3.5\lib\site-packages\openne-0.0.0-py3.5.egg\openne_main.py", line 182, in
main(parse_args())
File "D:\software\Python3.5\lib\site-packages\openne-0.0.0-py3.5.egg\openne_main.py", line 137, in main
g.read_node_label(args.label_file)
File "D:\software\Python3.5\lib\site-packages\openne-0.0.0-py3.5.egg\openne\graph.py", line 89, in read_node_label
self.G.nodes[vec[0]]['label'] = vec[1:]
File "D:\software\Python3.5\lib\site-packages\networkx\classes\reportviews.py", line 178, in getitem
return self._nodes[n]
KeyError: '703'
I/O error(2): No such file or directory while evaluating method tadw
Traceback (most recent call last):
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydevd.py", line 1741, in
main()
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/liujinxin/Desktop/EvalNE-master/examples/simple-example.py", line 43, in
edge_embedding_methods=edge_emb, input_delim=' ', output_delim=' ')
File "D:\software\Python3.5\lib\site-packages\evalne\evaluation\evaluator.py", line 695, in evaluate_cmd
input_delim, output_delim, write_weights, write_dir, verbose)
File "D:\software\Python3.5\lib\site-packages\evalne\evaluation\evaluator.py", line 744, in _evaluate_ne_cmd
num_vectors = sum(1 for _ in open(tmpemb))
FileNotFoundError: [Errno 2] No such file or directory: './emb.tmp'
Backend TkAgg is interactive backend. Turning interactive mode on.
Failed to enable GUI event loop integration for 'tk'
Traceback (most recent call last):
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydev_ipython\matplotlibtools.py", line 31, in do_enable_gui
enable_gui(guiname)
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydev_ipython\inputhook.py", line 536, in enable_gui
return gui_hook(app)
File "D:\software\pycharm\PyCharm Community Edition 2018.3.5\helpers\pydev\pydev_ipython\inputhook.py", line 285, in enable_tk
app = TK.Tk()
File "D:\software\Python3.5\lib\tkinter_init.py", line 1877, in init
self.tk = _tkinter.create(screenName, baseName, className, interactive, wantobjects, useTk, sync, use)
_tkinter.TclError: Can't find a usable init.tcl in the following directories:
D:/software/Python3.5/lib/tcl8.6 D:/software/lib/tcl8.6 D:/lib/tcl8.6 D:/software/library D:/library D:/tcl8.6.4/library D:/tcl8.6.4/library
This probably means that Tcl wasn't installed properly.
Hi,
Is there a way ti use your package for custom network embedding method ?
Thank you.
I am trying to install EvalNE v0.3.2 on python 3.6.12 on macOS and execute simple_example.py.
While executing
$conda install --file requirements.txt
the output is
error: numpy 1.15.1 is installed but numpy>=1.15.4 is required by {'pandas'}
Basically, it can be reproduced by following the installation guide:
$git clone https://github.com/Dru-Mara/EvalNE.git
$cd EvalNE
$pip3 install -r requirements.txt
$sudo python3 setup.py install
or
$git clone https://github.com/Dru-Mara/EvalNE.git
$cd EvalNE
$conda install --file requirements.txt
$sudo python3 setup.py install
By ignoring the error and later running
$cd examples/
$python3 simple_example.py
we obtain
evalne/utils/preprocess.py", line 523, in prep_graph G.remove_edges_from(G.selfloop_edges()) AttributeError: 'Graph' object has no attribute 'selfloop_edges'
I think the issue is that we need to specify the versions in the requirements more clearly, i.e. the versions of scipy, pandas, tqdm and matplotlib.
(EvalNE) C:\Users\13688\Downloads\EvalNE-master\examples\api_examples>python simple-example.py
Traceback (most recent call last):
File "", line 1, in
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "D:\Software\MiniConda\envs\EvalNE\lib\runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "D:\Software\MiniConda\envs\EvalNE\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "D:\Software\MiniConda\envs\EvalNE\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\13688\Downloads\EvalNE-master\examples\api_examples\simple-example.py", line 33, in
result = nee.evaluate_baseline(method=method)
File "D:\Software\MiniConda\envs\EvalNE\lib\site-packages\evalne\evaluation\evaluator.py", line 301, in evaluate_baseline
train_pred, test_pred = util.run_function(timeout, _eval_sim,
File "D:\Software\MiniConda\envs\EvalNE\lib\site-packages\evalne\utils\util.py", line 189, in run_function
p.start()
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\popen_spawn_win32.py", line 45, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "D:\Software\MiniConda\envs\EvalNE\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
Hi Alex,
I noticed this bug when I wanted to use the Score class separately from any other class in the evalne package, simply because it allows you to easily calculate and plot performance metrics. However, when I wanted to use the precisionatk
function in evalne/evaluation/score.py at line 598 I got the following error:
TypeError: 'zip' object is not subscriptable
Current solution: I solved it by first encapsulating the zip-object in a list-call as such list(zip(*self._sorted))[0]
.
Best,
Pieter-Paul
Hi Alexandru,
I don't fully understand how network reconstruction is done.
For example, what is the difference between LPEvaluator and NREvaluator when train_frac is the same number, for example 0.7 ?
Best regards.
Is this correct?
Every positive training edge exists in the training graph, but none of the positive testing edges exist in the training graph.
If so, I'm not sure I understand the reasoning. Isn't the goal to predict missing edges? If the training edges exist in the training graph then they're not missing.
In particular, methods which return very high scores for edges which already exist will bias the training. For example, I'm using quasi-local methods like superposed random walks (just running a few random walks of length 3 and then adding the results). If an edge already exists between two nodes then this superposed score will tend to be very very high. As a result, my test scores have very high precision but very low recall. This makes sense to me if the classifier is learning "very very high SRW score => edge".
Am I misinterpreting something here?
Dear authors,
I try to understand a procedure around train/test splitting for evaluating network embeddings on link prediction you implemented in your package. I posted a similar question yesterday on Data Science Stack Exchange.
If I understand your paper correctly, the node embedding is performed only on positive examples (edges) on the training set; false examples are used only to train a classifier? Is this correct?
Thanks.
Best, Andrej
Hello,
is this framework suitable with RDF data, since RDF induces a multigraph?
Describe the bug
I have installed EvalNE, OpenNE library, PRUNE and Metapath2Vec following the instructions.
When I run evaluator_example.py, I encounter several errors and warnings.
metapath2vec++
contains 2 more lines than expected. Will consider them part of the header and ignore them... Expected num_lines 703, obtained lines 705.To Reproduce
Steps to reproduce the error:
cd examples/
python3 evaluator_example.py
Error 1
Traceback (most recent call last):
File "evaluator_example.py", line 185, in
main()
File "evaluator_example.py", line 67, in main
eval_other(nee, scoresheet)
File "evaluator_example.py", line 153, in eval_other
for res in results:
TypeError: 'Results' object is not iterable
Error 2
Traceback (most recent call last):
File "/home/huangxk/workspace_python/embedding/EvalNE/examples/evaluator_example.py", line 185, in
main()
File "/home/huangxk/workspace_python/embedding/EvalNE/examples/evaluator_example.py", line 70, in main
scoresheet.write_tabular(filename=os.path.join(outpath, 'eval_output.txt'), metric='auroc')
File "/home/huangxk/workspace_python/embedding/EvalNE/venv_for_evlne/lib/python3.6/site-packages/evalne/evaluation/score.py", line 204, in write_tabular
df.to_csv(f, sep='\t', na_rep='NA')
File "/home/huangxk/workspace_python/embedding/EvalNE/venv_for_evlne/lib/python3.6/site-packages/pandas/core/generic.py", line 3228, in to_csv
formatter.save()
File "/home/huangxk/workspace_python/embedding/EvalNE/venv_for_evlne/lib/python3.6/site-packages/pandas/io/formats/csvs.py", line 202, in save
self._save()
File "/home/huangxk/workspace_python/embedding/EvalNE/venv_for_evlne/lib/python3.6/site-packages/pandas/io/formats/csvs.py", line 310, in _save
self._save_header()
File "/home/huangxk/workspace_python/embedding/EvalNE/venv_for_evlne/lib/python3.6/site-packages/pandas/io/formats/csvs.py", line 278, in _save_header
writer.writerow(encoded_labels)
TypeError: a bytes-like object is required, not 'str'
Error 3
Preprocessing graph...
Repetition 0 of experiment
Evaluating baselines...
Evaluating Embedding methods...
ERROR:root:No test edges in trainvalid_split. Recomputing correct split...
Running command...
Warning 4
WARNING:root:Output of method metapath2vec++ contains 2 more lines than expected. Will consider them part of the header and ignore them... Expected num_lines 703, obtained lines 705.
WARNING:root:Output provided by method metapath2vec++
contains 129 columns, 128 expected! Taking first column as nodeID...
WARNING:root:Output of method node2vec
contains 1 more lines than expected. Will consider them part of the header and ignore them... Expected num_lines 703, obtained lines 704.
WARNING:root:Output provided by method node2vec
contains 129 columns, 128 expected! Taking first column as nodeID...
WARNING:root:Output of method deepwalk
contains 1 more lines than expected. Will consider them part of the header and ignore them... Expected num_lines 703, obtained lines 704.
WARNING:root:Output provided by method deepwalk
contains 129 columns, 128 expected! Taking first column as nodeID...
My solutions
I have tried to solve Error 1 and Error 2 and it works (but i am not sure whether it is the right solution). Although Error 3 and Warning 4 are WARNING, I want to know the reason and whether I should ignore them or not.
My solution to TypeError: 'Results' object is not iterable.
In file evaluator_example.py, line 149-150, line 173-174 :
for res in results:
scoresheet.log_results(res)
change them to
scoresheet.log_results(results)
My solution to TypeError: a bytes-like object is required, not 'str'
It seems that this error is caused by text/binary mode. This question in stackoverflow may be helpful. So I tried to change the source code of evalne: in file evalne/evaluation/score.py, line 201-202
f = open(filename, 'a+b')
f.write(header.encode())
change them to
f = open(filename, 'a')
f.write(header)
ERROR:root:No test edges in trainvalid_split. Recomputing correct split...
Why this error message? Should I ignore it?
Warning 4
It seems that they are related to OpenNE?
Result
Even with Error 3 and Warning 4, I still get result file in example/output/eval_output.txt
Evaluation results (auroc):
-----------------------
network
random_prediction 0.4942
common_neighbours 0.8458
jaccard_coefficient 0.7255
adamic_adar_index 0.8551
preferential_attachment 0.9376
resource_allocation_index 0.853
PRUNE 0.8299
metapath2vec++ 0.8218
node2vec 0.8796
deepwalk 0.8603
line 0.8997
Is this correct?
Desktop (please complete the following information):
Thanks for sharing this great library. I am learning to use it.
Best,
Xikun
IOError: [Errno 2] No such file or directory: './emb.tmp' while running python evalne ./examples/conf_parTest.ini
Hi,
Do you think your code can also be applied for the evaluation of heterogeneous and/or mutiplex network embedding ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.