wmayner / pyemd Goto Github PK
View Code? Open in Web Editor NEWFast EMD for Python: a wrapper for Pele and Werman's C++ implementation of the Earth Mover's Distance metric
License: MIT License
Fast EMD for Python: a wrapper for Pele and Werman's C++ implementation of the Earth Mover's Distance metric
License: MIT License
When I run pip install pyemd
, it's erroring out on x86_64-linux-gnu-gcc: error: pyemd/emd.c: No such file or directory
. Is there some external dependency? Thanks.
hi
just a humble suggestion to use multiple processes for number of trials in CEEMDAN.
Hi, the ICCV paper makes use of a thresholded ground distance d_t(a,b) = min(t, d(a,b)) to better the time complexity, does this need to be reflected in the distance_matrix before passing it as an argument? If so what value do we set for the threshold?
Thanks for providing a wrapper to the emd calculation in python:
what is the meaning of the signature? as far as i know, emd basically calculate a distance between two distribution. I guess that the distance_matrix as input is the ground distance [e.g. pairwise L2 distance], but what does the signature stands for? is it considering the weight in the historgram?
How about the N? looks like that it is a total number of bins in both histogram.
Thanks for your time in answering the question.
Edwin
pyemd fails installing on python 3.7 environment on Mac OS X Mojave (version 10.14.3).
conda create -n test_pyemd_py36 python=3.6
pip install pyemd
Installation successful.
conda create -n test_pyemd_py37 python=3.7
pip install premed
Fails while rebuilding wheels (because not published).
Error log:
ERROR: Complete output from command /anaconda3/envs/test_pyemd_py37/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/d1/fh2k9h6s6j39smdvttzv4hx40000gn/T/pip-install-06if_qq4/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/d1/fh2k9h6s6j39smdvttzv4hx40000gn/T/pip-record-9l7e48vz/install-record.txt --single-version-externally-managed --compile:
ERROR: running install
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.7
creating build/lib.macosx-10.7-x86_64-3.7/pyemd
copying pyemd/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/pyemd
copying pyemd/__about__.py -> build/lib.macosx-10.7-x86_64-3.7/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.macosx-10.7-x86_64-3.7
creating build/temp.macosx-10.7-x86_64-3.7/pyemd
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/envs/test_pyemd_py37/include -arch x86_64 -I/anaconda3/envs/test_pyemd_py37/include -arch x86_64 -I/anaconda3/envs/test_pyemd_py37/include/python3.7m -I/anaconda3/envs/test_pyemd_py37/lib/python3.7/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.macosx-10.7-x86_64-3.7/pyemd/emd.o
warning: include path for stdlibc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
pyemd/emd.cpp:556:10: fatal error: 'utility' file not found
#include <utility>
^~~~~~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command "/anaconda3/envs/test_pyemd_py37/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/private/var/folders/d1/fh2k9h6s6j39smdvttzv4hx40000gn/T/pip-install-06if_qq4/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/d1/fh2k9h6s6j39smdvttzv4hx40000gn/T/pip-record-9l7e48vz/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/d1/fh2k9h6s6j39smdvttzv4hx40000gn/T/pip-install-06if_qq4/pyemd/
Thanks for the project!
I've been comparing the results between pyemd and PythonOT/POT and I found inconsistencies in the results. Not sure if I am applying it in a wrong way.
numpy 1.20.3
scipy 1.8.0
ot 0.8.2
pyemd 0.5.1
Here is the code to reproduce it:
import numpy as np
import ot
import pyemd
from scipy.spatial import distance_matrix
np.random.seed(0)
n = 3
X = np.random.random((n, 2))
Y = np.random.random((n, 2))
D = distance_matrix(X, Y)
d0 = np.ones(n) / n
d1 = np.ones(n) / n
e, flow = emd_with_flow(d0, d1, D)
print(e, flow)
Output:
(0.0, [[0.333333, 0.0, 0.0], [0.0, 0.333333, 0.0], [0.0, 0.0, 0.333333]])
In fact, the earth mover distance is usually 0 for histograms that are uniform.
Here with POT:
e = ot.emd2(d0, d1, D)
flow = ot.emd(d0, d1, D)
print(e, flow.tolist())
Output:
0.31591990155991395 [[0.0, 0.0, 0.3333333333333333], [0.0, 0.3333333333333333, 0.0], [0.3333333333333333, 0.0, 0.0]]
The flow is also different in this case.
Am I missing something or is this some issue with the code in either repository?
pyemd in databricks could not be installed in Databrick Runtime version 4.2 (no workers). The error is below:
%sh
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz
Stepping: 1
CPU MHz: 2294.685
BogoMIPS: 4589.37
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 51200K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt
Python:
sys.version_info
Out[25]: sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
Pip:
%sh
/databricks/python3/bin/python3 -m pip freeze
ansi2html==1.1.1
azure-common==1.1.18
azure-nspkg==3.0.2
azure-storage-blob==1.1.0
azure-storage-common==1.1.0
azure-storage-nspkg==3.1.0
backports.shutil-get-terminal-size==1.0.0
boto==2.42.0
boto3==1.4.1
botocore==1.4.70
brewer2mpl==1.4.1
certifi==2016.2.28
cffi==1.7.0
chardet==2.3.0
colorama==0.3.7
configobj==5.0.6
cryptography==1.5
cycler==0.10.0
Cython==0.24.1
decorator==4.0.10
docutils==0.14
enum34==1.1.6
et-xmlfile==1.0.1
freetype-py==1.0.2
funcsigs==1.0.2
fusepy==2.0.4
ggplot==0.6.8
html5lib==0.999
idna==2.1
ipaddress==1.0.16
ipython==2.2.0
ipython-genutils==0.1.0
jdcal==1.2
Jinja2==2.8
jmespath==0.9.0
llvmlite==0.13.0
lxml==3.6.4
MarkupSafe==0.23
matplotlib==1.5.3
mpld3==0.2
msgpack-python==0.4.7
ndg-httpsclient==0.3.3
numba==0.28.1
numpy==1.16.1
openpyxl==2.3.2
pandas==0.24.1
pathlib2==2.1.0
patsy==0.4.1
pexpect==4.0.1
pickleshare==0.7.4
Pillow==3.3.1
pkg-resources==0.0.0
ply==3.9
prompt-toolkit==1.0.7
psycopg2==2.6.2
ptyprocess==0.5.1
py4j==0.10.3
pyarrow==0.8.0
pyasn1==0.1.9
pycparser==2.14
pycurl==7.43.0
Pygments==2.1.3
pygobject==3.20.0
pyOpenSSL==16.0.0
pyparsing==2.2.0
pypng==0.0.18
python-apt==1.1.0b1+ubuntu0.16.4.1
python-dateutil==2.5.3
python-geohash==0.8.5
pytz==2016.6.1
requests==2.11.1
s3transfer==0.1.9
scikit-learn==0.18.1
scipy==0.18.1
scour==0.32
seaborn==0.7.1
simplejson==3.8.2
singledispatch==3.4.0.3
six==1.10.0
ssh-import-id==5.5
statsmodels==0.8.0
traitlets==4.3.0
unattended-upgrades==0.1
urllib3==1.19.1
virtualenv==15.0.1
wcwidth==0.1.7
Error:
%sh
/databricks/python3/bin/python3 -m pip install pyemd
Downloading https://files.pythonhosted.org/packages/c0/c5/7fea8e7a71cd026b30ed3c40e4c5ea13a173e28f8855da17e25271e8f545/pyemd-0.5.1.tar.gz (91kB)
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/pyemd.egg-info
writing requirements to pip-egg-info/pyemd.egg-info/requires.txt
writing pip-egg-info/pyemd.egg-info/PKG-INFO
writing top-level names to pip-egg-info/pyemd.egg-info/top_level.txt
writing dependency_links to pip-egg-info/pyemd.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/pyemd.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-eii9t9ra/pyemd/setup.py", line 121, in <module>
'Programming Language :: Python :: 3.6',
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/__init__.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 278, in run
self.find_sources()
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 293, in find_sources
mm.run()
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 524, in run
self.add_defaults()
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/egg_info.py", line 560, in add_defaults
sdist.add_defaults(self)
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/[REDACTED]/python3/lib/python3.5/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/usr/lib/python3.5/distutils/cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/lib/python3.5/distutils/cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "/tmp/pip-install-eii9t9ra/pyemd/setup.py", line 77, in finalize_options
import numpy
ImportError: No module named 'numpy'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-eii9t9ra/pyemd/
Might it be that the path to python in /usr/lib or similar is hardcoded somewhere there?
The link to the code is broken. The correct link is:
http://www.ariel.ac.il/sites/ofirpele/fastemd/code/
Hello,
My question may be a very easy one but I am loosing my nerves trying to solve it.
here are my parameters :
first_histogram = [1. 1. 1.]
second_histogram = [2. 0. 0.]
distance_matrix = array([[0. , 0. , 0. , 0.60058105, 1. ],
[0. , 0. , 0. , 0.60058105, 1. ],
[0. , 0. , 0. , 0.60058105, 1. ],
[0.60058105, 0.60058105, 0.60058105, 0. , 0.98793931],
[1. , 1. , 1. , 0.98793931, 0. ]])
(My distance matrix is the result of sklearn.metrics.pairwise.cosine_distances(), so it truly is a distance matrix)
Now if I try to do :
D_EMD = emd(first_histogram, second_histogram, distance_matrix)
The code runs for ever without getting any results, without any Error Raised...
Does anyone have any idea what I'm doing wrong?
Thanks a lot !
Christel
Hello,
I believe users would find it useful to have a built in method for calculating the EMD statistic on two arrays without having to build the histograms and distance matrix.
I've written a simple function to do this myself - I'm happy to write it up properly and submit a pull request if you're happy to incorporate it into the API.
from pyemd import emd
import numpy as np
from sklearn.metrics.pairwise import pairwise_distances
def array_emd(x, y, bins=30):
xy_min = min(np.min(x), np.min(y))
xy_max = max(np.max(x), np.max(y))
x_hist, bin_edges = np.histogram(x, range=(xy_min, xy_max), bins=bins)
y_hist, _ = np.histogram(y, range=(xy_min, xy_max), bins=bins)
bin_middles = np.mean([bin_edges[:-1], bin_edges[1:]], axis=0)
x_hist = x_hist.astype(dtype="float64")
y_hist = y_hist.astype(dtype="float64")
dist_mat = pairwise_distances(bin_middles.reshape(-1, 1))
return emd(x_hist, y_hist, dist_mat)
I am using pipenv with python 3.9
Here's the error I get
Error: An error occurred while installing pyemd!
ERROR: Command errored out with exit status 1:
command: /home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h88kc0oe/pyemd/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h88kc0oe/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ilv1c_c6
cwd: /tmp/pip-install-h88kc0oe/pyemd/
Complete output (20 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/pyemd
copying pyemd/__about__.py -> build/lib.linux-x86_64-3.9/pyemd
copying pyemd/__init__.py -> build/lib.linux-x86_64-3.9/pyemd
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/pyemd
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/include -I/usr/include/python3.9 -I/home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/lib/python3.9/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-3.9/pyemd/emd.o
pyemd/emd.cpp:23:10: fatal error: Python.h: No such file or directory
23 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for pyemd
ERROR: Command errored out with exit status 1:
command: /home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h88kc0oe/pyemd/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h88kc0oe/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-xe7zj_p1/install-record.txt --single-version-externally-managed --compile --install-headers /home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/include/site/python3.9/pyemd
cwd: /tmp/pip-install-h88kc0oe/pyemd/
Complete output (20 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/pyemd
copying pyemd/__about__.py -> build/lib.linux-x86_64-3.9/pyemd
copying pyemd/__init__.py -> build/lib.linux-x86_64-3.9/pyemd
warning: build_py: byte-compiling is disabled, skipping.
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/pyemd
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/include -I/usr/include/python3.9 -I/home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/lib/python3.9/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-3.9/pyemd/emd.o
pyemd/emd.cpp:23:10: fatal error: Python.h: No such file or directory
23 | #include "Python.h"
| ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h88kc0oe/pyemd/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h88kc0oe/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-xe7zj_p1/install-record.txt --single-version-externally-managed --compile --install-headers /home/perseus/.local/share/virtualenvs/similarity-_ln239NJ/include/site/python3.9/pyemd Check the logs for full command output.
Hi,
I get an error shown as follow:
pyemd/emd.pyx in pyemd.emd.emd (pyemd/emd.cpp:2045)()
pyemd/emd.pyx in pyemd.emd.validate (pyemd/emd.cpp:1769)()
ValueError: Signature dimension cannot be larger than dimensions of distance matrix
the dimension of my variables are shown here:
first_signature.shape = (363, 363)
second_signature.shape = (363,)
distance_matrix.shape = (363,)
What does the error means?
But it exists on PyPI: https://pypi.org/project/pyemd/1.0.0/
Hi, I'd appreciate some explanation on what the inputs/outputs actually mean. I see the distance_matrix as the matrix of distances between the cluster centroids/representatives as described in the paper, where element (i,j) is the distance between cluster i from first_histogram and cluster j from second_histogram. In this case there is of course no restriction on distance_matrix being symmetric. However, I see that in the examples the matrix is always symmetric. Is this just a coincidence or is there something I don't know?
Also, why is it that the following two examples produce different results? Why is it that the flow is the same?
first_histogram = np.array([4.0, 6.0])
second_histogram = np.array([5.0, 5.0])
distance_matrix = np.array([[0.5, 0.0],[0.0, 0.5]])
emd_with_flow(first_histogram, second_histogram, distance_matrix)
(0.0, [[4.0, 0.0], [1.0, 5.0]])
first_histogram = np.array([4.0, 6.0])
second_histogram = np.array([5.0, 5.0])
distance_matrix = np.array([[0.0, 0.5],[0.5, 0.0]])
emd_with_flow(first_histogram, second_histogram, distance_matrix)
(0.5, [[4.0, 0.0], [1.0, 5.0]])
Thanks for your time.
I was running the function imd_sample and I've noticed that the function returns a result even when the arrays are not 1D.
Thanks for the wrapper, it has been very useful to me! For my application, I would like to get the flows as well, and I was wondering whether it would be possible?
I read over the original C++ code, and if I understood well you can pass a pointer to a std::vector<std::vector<NUM_T>>
initialised with 0, and it will modify the container in place, giving you the flows. Am I right?
I am a complete beginners with Cython. Would you have any pointer to some useful tutorials (besides the doc) to get me started? Since python functions do not modify objects in place, I am a bit confused.
I'll see what I can do, and file a PR once/if I ever managed to include this function!
Hi, I'm facing this error message when I tried to import pyemd in my code:
Traceback (most recent call last):
File "vincent.py", line 30, in <module>
from pyemd import emd
File "/home/cragkhit/anaconda3/envs/myenv/lib/python3.6/site-packages/pyemd/__init__.py", line 67, in <module>
from .emd import emd
ImportError: /home/cragkhit/anaconda3/envs/myenv/lib/python3.6/site-packages/pyemd/emd.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E
I use Anaconda virtual environment in my development. The error message only occur on Ubuntu. Everything works fine on my macOS. I'm using
Distributor ID: Ubuntu
Description: Ubuntu 16.04.2 LTS
Release: 16.04
Codename: xenial
I have an installation issue. I am using Python 3.7.3 on Windows 10. Any suggestions?
command: 'c:\users\walmar2\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\WALMAR2\\AppData\\Local\\Temp\\pip-install-_iqwmdp1\\pyemd\\setup.py'"'"'; __file__='"'"'C:\\Users\\WALMAR2\\AppData\\Local\\Temp\\pip-install-_iqwmdp1\\pyemd\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\WALMAR2\AppData\Local\Temp\pip-wheel-y42hosf4' --python-tag cp37
cwd: C:\Users\WALMAR2\AppData\Local\Temp\pip-install-_iqwmdp1\pyemd\
Complete output (17 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-3.7
creating build\lib.win-amd64-3.7\pyemd
copying pyemd\__about__.py -> build\lib.win-amd64-3.7\pyemd
copying pyemd\__init__.py -> build\lib.win-amd64-3.7\pyemd
running build_ext
building 'pyemd.emd' extension
creating build\temp.win-amd64-3.7
creating build\temp.win-amd64-3.7\Release
creating build\temp.win-amd64-3.7\Release\pyemd
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.21.27702\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT -Ipyemd -Ic:\users\walmar2\appdata\local\programs\python\python37\include -Ic:\users\walmar2\appdata\local\programs\python\python37\include -Ic:\users\walmar2\appdata\local\programs\python\python37\lib\site-packages\numpy\core\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.21.27702\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" /EHsc /Tppyemd/emd.cpp /Fobuild\temp.win-amd64-3.7\Release\pyemd/emd.obj
emd.cpp
c:\users\walmar2\appdata\local\programs\python\python37\include\pyconfig.h(203): fatal error C1083: Datei (Include) kann nicht ge”ffnet werden: "basetsd.h": No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.21.27702\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Failed building wheel for pyemd
The Debian package of pyemd fails to build on the 32-bit powerpc port because the tests time out after 150 minutes of inactivity, while on other platforms including ppc64el and ppc64 they complete in under half a second. I logged into the powerpc porterbox perotto.debian.net, executed a build and noticed that the Python process running the tests uses 100% CPU. I deleted tests one at a time until the remaining tests were passing. Only three tests were hanging with 100% CPU: test_emd_3
test_emd_with_flow_3
test_emd_with_flow_4
. The three hanging tests seem to have one thing in common, they all have a distance_matrix
containing only zeros. One other test that passes has this feature too test_emd_with_flow_5
though. The three hanging tests also seem to have similar signature parameters. When running Python in gdb, interrupting the process gives a stack trace that indicates the compute_shortest_path
function from min_cost_flow.hpp
is using the CPU.
Any ideas on how to debug this?
ERROR: Failed building wheel for pyemd
Running setup.py clean for pyemd
Failed to build pyemd
Installing collected packages: pyemd
Running setup.py install for pyemd ... error
ERROR: Command errored out with exit status 1:
command: 'D:\tfs\pandit\anaconda\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\mukesh.sah\AppData\Local\Temp\pip-install-db68s9hb\pyemd\setup.py'"'"'; file='"'"'C:\Users\mukesh.sah\AppData\Local\Temp\pip-install-db68s9hb\pyemd\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\mukesh.sah\AppData\Local\Temp\pip-record-swz_qpxc\install-record.txt' --single-version-externally-managed --compile --install-headers 'D:\tfs\pandit\anaconda\Include\pyemd'
cwd: C:\Users\mukesh.sah\AppData\Local\Temp\pip-install-db68s9hb\pyemd
Complete output (11 lines):
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.7
creating build\lib.win-amd64-3.7\pyemd
copying pyemd_about_.py -> build\lib.win-amd64-3.7\pyemd
copying pyemd_init_.py -> build\lib.win-amd64-3.7\pyemd
running build_ext
building 'pyemd.emd' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
----------------------------------------
ERROR: Command errored out with exit status 1: 'D:\tfs\pandit\anaconda\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\mukesh.sah\AppData\Local\Temp\pip-install-db68s9hb\pyemd\setup.py'"'"'; file='"'"'C:\Users\mukesh.sah\AppData\Local\Temp\pip-install-db68s9hb\pyemd\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\mukesh.sah\AppData\Local\Temp\pip-record-swz_qpxc\install-record.txt' --single-version-externally-managed --compile --install-headers 'D:\tfs\pandit\anaconda\Include\pyemd' Check the logs for full command output.
I am getting this error though I have installed Microsoft Visual C++ 14.0.
I am using anaconda version 2020.02
python 3.7.6
pymed version = 0.5.1
please help to fix this issue. I am installing through pip.
no version is working fine wheel is not stable
Hi, wmayner, thanks for your wonderful code! The speed of this python wrapper is very fast. However, I find the original C++ code from Ofir Pele is much slower than this python wrapper and I do not know why this happen. I run the C++ code on VS 2013. Would you please help me to find the reason? Did you modify the original C++ code? Thanks very much!
Hi,
Do you plan on adding support for multi-dimensional points ? I am working with (x,y,z) coordinates and it would be of huge help if this support would be added.
Thanks!
Thanks for creating this wrapper! I'm trying to use it for an fMRI meta-analysis, but am running into memory overload problems. Do you by chance know the approximate memory complexity of this algorithm?
I currently get a segmentation fault when i try to use pyemd. Can you give any hints as to what might produce this from Python ?
Took a look at https://github.com/wmayner/pyemd/blob/develop/pyemd/lib/emd_hat_impl.hpp but not sure why it produces the Segmentation fault
Program received signal SIGSEGV, Segmentation fault. emd_hat_impl<double, (FLOW_TYPE_T)0>::operator() (this=this@entry=0x7fffffffd07f, POrig=..., QOrig=..., P=..., Q=..., C=..., extra_mass_penalty=extra_mass_penalty@entry=-1, F=F@entry=0x0) at pyemd/lib/emd_hat_impl.hpp:435 435 pyemd/lib/emd_hat_impl.hpp: No such file or directory
Excuse me, does pyemd package include the EEMD algorithm?
Hi, not necessarily an issue, what precisely is the distance matrix required? Thanks.
if I want to add weights and the signature size is different, what can I do?
Hi! Thanks a lot for the project. Is there any plan to support python 3.11?
When I try to install the package there I get the following error:
Collecting pyemd
Using cached pyemd-0.5.1.tar.gz (91 kB)
Preparing metadata (setup.py) ... done
Collecting numpy<2.0.0,>=1.9.0
Using cached numpy-1.23.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Building wheels for collected packages: pyemd
Building wheel for pyemd (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
/home/kyriakos/miniconda3/envs/valentine11/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/pyemd
copying pyemd/__init__.py -> build/lib.linux-x86_64-cpython-311/pyemd
copying pyemd/__about__.py -> build/lib.linux-x86_64-cpython-311/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/pyemd
gcc -pthread -B /home/kyriakos/miniconda3/envs/valentine11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/kyriakos/miniconda3/envs/valentine11/include -fPIC -O2 -isystem /home/kyriakos/miniconda3/envs/valentine11/include -fPIC -I/home/kyriakos/miniconda3/envs/valentine11/include/python3.11 -I/tmp/pip-install-hch1ioh_/pyemd_850fda2363c24de38b5a817d08842ae6/.eggs/numpy-1.23.4-py3.11-linux-x86_64.egg/numpy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-cpython-311/pyemd/emd.o
pyemd/emd.cpp:200:12: fatal error: longintrepr.h: No such file or directory
200 | #include "longintrepr.h"
| ^~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyemd
Running setup.py clean for pyemd
Failed to build pyemd
Installing collected packages: numpy, pyemd
Running setup.py install for pyemd ... error
error: subprocess-exited-with-error
× Running setup.py install for pyemd did not run successfully.
│ exit code: 1
╰─> [20 lines of output]
running install
/home/kyriakos/miniconda3/envs/valentine11/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/pyemd
copying pyemd/__init__.py -> build/lib.linux-x86_64-cpython-311/pyemd
copying pyemd/__about__.py -> build/lib.linux-x86_64-cpython-311/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/pyemd
gcc -pthread -B /home/kyriakos/miniconda3/envs/valentine11/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/kyriakos/miniconda3/envs/valentine11/include -fPIC -O2 -isystem /home/kyriakos/miniconda3/envs/valentine11/include -fPIC -I/home/kyriakos/miniconda3/envs/valentine11/include/python3.11 -I/home/kyriakos/miniconda3/envs/valentine11/lib/python3.11/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-cpython-311/pyemd/emd.o
pyemd/emd.cpp:200:12: fatal error: longintrepr.h: No such file or directory
200 | #include "longintrepr.h"
| ^~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> pyemd
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
When trying to upgrade to 0.5.0 I'm getting the following output:
Collecting pyemd
Using cached pyemd-0.5.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-odst_5et/pyemd/setup.py", line 87, in <module>
README = f.read()
File "/srv/.pyenv/versions/3.6.3/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2247: ordinal not in range(128)
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-odst_5et/pyemd/
Seems like there's an issue with decoding a file. Shouldn't unicode be used instead of ascii? Is this an issue with my system configuration?
Thanks!
Note: I was able to solve the problem by installing from source. I'm posting just to document the issue.
I'm running python3.5 with the most recent distribution of anaconda
~ $ python --version
Python 3.5.0 :: Anaconda 2.4.0 (x86_64)
I get the following error when I try to install premed
~ $ pip install --no-cache-dir pyemd
Collecting pyemd
Downloading pyemd-0.2.0.tar.gz (52kB)
100% |████████████████████████████████| 53kB 4.6MB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 20, in <module>
File "/private/var/folders/z4/8yrkxb250h3d9psym1bmqvgw0000gw/T/pip-build-g8fo_tbk/pyemd/setup.py", line 44, in <module>
extensions = cythonize(extensions)
File "/Users/fred/anaconda/lib/python3.5/site-packages/Cython/Build/Dependencies.py", line 758, in cythonize
aliases=aliases)
File "/Users/fred/anaconda/lib/python3.5/site-packages/Cython/Build/Dependencies.py", line 651, in create_extension_list
for file in nonempty(sorted(extended_iglob(filepattern)), "'%s' doesn't match any files" % filepattern):
File "/Users/fred/anaconda/lib/python3.5/site-packages/Cython/Build/Dependencies.py", line 103, in nonempty
raise ValueError(error_msg)
ValueError: 'pyemd/emd.pyx' doesn't match any files
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/z4/8yrkxb250h3d9psym1bmqvgw0000gw/T/pip-build-g8fo_tbk/pyemd
I can install fine using the source on github. I can also install from pip using the standard python 3.4 distribution (haven't tried the standard python 3.5)
Hey there :)
First, thank you very much for PyEMD 👍
I'm trying to install using pip with python3.6 under RHEL 7 but I get the following error. This is consistent across several machines / installations I'm trying to build on.
Does someone has any idea where does this come from ?
Any help greatly appreciated !
# pip install pyemd
Collecting pyemd
Using cached https://files.pythonhosted.org/packages/c0/c5/7fea8e7a71cd026b30ed3c40e4c5ea13a173e28f8855da17e25271e8f545/pyemd-0.5.1.tar.gz
Requirement already satisfied: numpy<2.0.0,>=1.9.0 in /usr/local/lib64/python3.6/site-packages (from pyemd) (1.16.4)
Installing collected packages: pyemd
Running setup.py install for pyemd ... error
ERROR: Complete output from command /usr/bin/python3.6 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-dchvj4ox/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-vvppeh96/install-record.txt --single-version-externally-managed --compile:
ERROR: running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/pyemd
copying pyemd/__about__.py -> build/lib.linux-x86_64-3.6/pyemd
copying pyemd/__init__.py -> build/lib.linux-x86_64-3.6/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/pyemd
gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.6m -I/usr/local/lib64/python3.6/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-3.6/pyemd/emd.o
pyemd/emd.cpp:23:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command "/usr/bin/python3.6 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-dchvj4ox/pyemd/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-vvppeh96/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-dchvj4ox/pyemd/
Assume, I have two array like below
first_signature = np.array([1.0, 2.0, 1.0, 2.0])
second_signature = np.array([2.0, 1.0, 2.0, 1.0])
How I calculate to this Distance matrix?
distance_matrix = np.array([[0.0, 1.0, 1.0, 2.0],
[1.0, 0.0, 2.0, 1.0],
[1.0, 2.0, 0.0, 1.0],
[2.0, 1.0, 1.0, 0.0]])
Collecting pyemd==0.4.4 (from -r production.txt (line 175))
Downloading https://files.pythonhosted.org/packages/34/61/0f2803463c695bdde498e63a6300f7878829427ecdb9144b6306883ce7f9/pyemd-0.4.4.tar.gz (67kB)
100% |████████████████████████████████| 71kB 10.5MB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-build-CQ7HD3/pyemd/setup.py", line 104, in
'Programming Language :: Python :: 3.6'
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 268, in init
self.fetch_build_eggs(attrs['setup_requires'])
File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 313, in fetch_build_eggs
replace_conflicting=True,
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 836, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1081, in best_match
return self.obtain(req, installer)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1093, in obtain
return installer(requirement)
File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 380, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 629, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 659, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 842, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1070, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1056, in run_setup
run_setup(setup_script, args)
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 240, in run_setup
raise
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 193, in setup_context
yield
File "/usr/lib/python2.7/contextlib.py", line 35, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 164, in save_modules
saved_exc.resume()
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 139, in resume
compat.reraise(type, exc, self._tb)
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 152, in save_modules
yield saved
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 193, in setup_context
yield
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 237, in run_setup
DirectorySandbox(setup_dir).run(runner)
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 267, in run
return func()
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 236, in runner
_execfile(setup_script, ns)
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 46, in _execfile
exec(code, globals, locals)
File "/tmp/easy_install-8c1Zng/numpy-1.17.0/setup.py", line 31, in
RuntimeError: Python version >= 3.5 required.
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-CQ7HD3/pyemd/
pyemd fails to install on python 3.7 environment on Mac OS X Catalina (version 10.15 beta).
similar to issue [#39] (#39)
I am using virtualenv with python 3.7.1.
pip install git+https://github.com/wmayner/pyemd/
Looking in indexes: https://pypi.python.org/simple, https://pypi.apple.com/simple
Collecting git+https://github.com/wmayner/pyemd/
Cloning https://github.com/wmayner/pyemd/ to /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9
Running command git clone -q https://github.com/wmayner/pyemd/ /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9
Building wheels for collected packages: pyemd
Building wheel for pyemd (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /Users/arian/Developer/workspace/signal/tilse/dev/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/setup.py'"'"'; __file__='"'"'/private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-wheel-y_qy4mk2 --python-tag cp37
cwd: /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/
Complete output (16 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.7
creating build/lib.macosx-10.7-x86_64-3.7/pyemd
copying pyemd/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/pyemd
copying pyemd/__about__.py -> build/lib.macosx-10.7-x86_64-3.7/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.macosx-10.7-x86_64-3.7
creating build/temp.macosx-10.7-x86_64-3.7/pyemd
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/arian/miniconda3/include -arch x86_64 -I/Users/arian/miniconda3/include -arch x86_64 -I/Users/arian/miniconda3/include/python3.7m -I/Users/arian/Developer/workspace/signal/tilse/dev/lib/python3.7/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.macosx-10.7-x86_64-3.7/pyemd/emd.o
clang: error: no such file or directory: 'pyemd/emd.cpp'
clang: error: no input files
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for pyemd
Running setup.py clean for pyemd
Failed to build pyemd
Installing collected packages: pyemd
Running setup.py install for pyemd ... error
ERROR: Command errored out with exit status 1:
command: /Users/arian/Developer/workspace/signal/tilse/dev/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/setup.py'"'"'; __file__='"'"'/private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-record-a05qi2c7/install-record.txt --single-version-externally-managed --compile --install-headers /Users/arian/Developer/workspace/signal/tilse/dev/include/site/python3.7/pyemd
cwd: /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/
Complete output (16 lines):
running install
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.7
creating build/lib.macosx-10.7-x86_64-3.7/pyemd
copying pyemd/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/pyemd
copying pyemd/__about__.py -> build/lib.macosx-10.7-x86_64-3.7/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.macosx-10.7-x86_64-3.7
creating build/temp.macosx-10.7-x86_64-3.7/pyemd
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/arian/miniconda3/include -arch x86_64 -I/Users/arian/miniconda3/include -arch x86_64 -I/Users/arian/miniconda3/include/python3.7m -I/Users/arian/Developer/workspace/signal/tilse/dev/lib/python3.7/site-packages/numpy/core/include -c pyemd/emd.cpp -o build/temp.macosx-10.7-x86_64-3.7/pyemd/emd.o
clang: error: no such file or directory: 'pyemd/emd.cpp'
clang: error: no input files
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/arian/Developer/workspace/signal/tilse/dev/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/setup.py'"'"'; __file__='"'"'/private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-req-build-e51_1oq9/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/p7/pynm4r6x1m1f4ww8g5j3lb3m0000gn/T/pip-record-a05qi2c7/install-record.txt --single-version-externally-managed --compile --install-headers /Users/arian/Developer/workspace/signal/tilse/dev/include/site/python3.7/pyemd Check the logs for full command output.
I am slightly confused by your description of emd and emd_samples, since for emd you wrote histogram as parameter and for emd_sample 1D samples.
If I want to compare two images by emd and I use emd_samples. Do I have to pass the images or the histograms of the images to the method?
When trying to import this package in python 2.7 I keep getting the same error:
>>> import pyemd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.linux-x86_64/egg/pyemd/__init__.py", line 63, in <module>
File "build/bdist.linux-x86_64/egg/pyemd/emd.py", line 7, in <module>
File "build/bdist.linux-x86_64/egg/pyemd/emd.py", line 6, in __bootstrap__
ImportError: /home/joris/.cache/Python-Eggs/pyemd-0.3.0-py2.7-linux-x86_64.egg-tmp/pyemd/emd.so: undefined symbol: _ZTINSt8ios_base7failureB5cxx11E
I can't seem to fix this, have tried reinstalling, installing using setup.py, dowloading all requirements (numpy, scipy, cython, g++/gcc, everything in requirements.txt, everything else I could think of) but nothing seems to work. What could cause this error?
It would be nice to switch to modern Python packaging standards at some point.
The Python community has standardised on pyproject.toml for Python packaging:
https://packaging.python.org/en/latest/tutorials/packaging-projects/
I haven't had time to complete this migration but I made a start here:
pip install pyemd
Collecting pyemd
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<pip.vendor.requebject at 0xf62b55cc>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pyemd/
Using cached pyemd-0.4.3.tar.gz
Requirement already satisfied: numpy<2.0.0,>=1.9.0 in ./anaconda3/lib/python3.6/site-packages (from pyemd)
Building wheels for collected packages: pyemd
Running setup.py bdist_wheel for pyemd ... error
Complete output from command /home/XXXXX/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-4hgf2w8e_);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmp0g4l273wpip-wheel- --
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/pyemd
copying pyemd/about.py -> build/lib.linux-x86_64-3.6/pyemd
copying pyemd/init.py -> build/lib.linux-x86_64-3.6/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/pyemd
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/wuxiaobo/anaconda3/include/python3.6m -Iy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-3.6/pyemd/emd.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from /home/XXXXX/anaconda3/include/python3.6m/Python.h:50:0,
from pyemd/emd.cpp:16:
/home/XXXXX/anaconda3/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/g
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
^
In file included from /home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1788:0,
from /home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from pyemd/emd.cpp:467:
/home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Usin_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
error: command 'gcc' failed with exit status 1
Failed building wheel for pyemd
Running setup.py clean for pyemd
Failed to build pyemd
Installing collected packages: pyemd
Running setup.py install for pyemd ... error
Complete output from command /home/XXXX/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-4hgf2ile__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-u3zpb6rz-record/compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/pyemd
copying pyemd/about.py -> build/lib.linux-x86_64-3.6/pyemd
copying pyemd/init.py -> build/lib.linux-x86_64-3.6/pyemd
running build_ext
building 'pyemd.emd' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/pyemd
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/XXXXX/anaconda3/include/python3.6m mpy/core/include -c pyemd/emd.cpp -o build/temp.linux-x86_64-3.6/pyemd/emd.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from /home/XXXXX/anaconda3/include/python3.6m/Python.h:50:0,
from pyemd/emd.cpp:16:
/home/XXXXX/anaconda3/include/python3.6m/pyport.h:686:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
^
In file included from /home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1788:0,
from /home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from pyemd/emd.cpp:467:
/home/XXXXX/anaconda3/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "UsNO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
error: command 'gcc' failed with exit status 1
@wmayner I would like to contribute to this project by adding the test for statistical information content of EMD components.
The test allows one to identify statistically significant IMFs in a noisy data.
The methodology was proposed by Zhaohua Wu, & Huang, N. E. (2004) in their paper, "A study of the characteristics of white noise using the empirical mode decomposition method" and has been established as a method to analyse trends especially for hydrological time series.
link: Original paper
Initially, only the residue component can be added, later on all IMFs can be added.
Kindly, let me know if the idea is in line with the aim of this project and if I may raise a PR.
Great module, or looks great anyway. Looking to use with Python 3.6.0 anaconda build, but no look through github clone install or pip install through github - can't import with error as in title. :(
do you have more challenging example than on main page
to how it works for complicated cases?
from pyemd import emd
import numpy as np
first_histogram = np.array([0.0, 1.0])
second_histogram = np.array([5.0, 3.0])
distance_matrix = np.array([[0.0, 0.5],
... [0.5, 0.0]])
emd(first_histogram, second_histogram, distance_matrix)
3.5
This is a reminder for issuing a new upstream release,
with a git tag and PyPI upload.
This will close at least the issue about Python 3.11 becase
it seems to just need running Cython to fix it.
The Debian package definitely builds fine with Python 3.11,
presumably because it runs Cython during the build process.
In addition it will fix the test failure with recent numpy.
PS: I've uploaded a snapshot of git master to Debian.
Hi All,
Sorry for this newbie question. I got this error after run pip install pyemd
:
building 'pyemd.emd' extension error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
I think I have installed VCC 14.0. This is my program screenshots:
How can I solve this problem?
Thank you very much.
Regards,
Nazmi
I'm having a bit of a problem trying to install this on my mac
(bio)pyemd mortonjt$ nosetests .
EE
======================================================================
ERROR: Failure: ImportError (dlopen(/Users/mortonjt/Documents/software/python/pyemd/pyemd/emd.cpython-35m-darwin.so, 2): Symbol not found: __ZNSt8__detail15_List_node_base7_M_hookEPS0_
Referenced from: /Users/mortonjt/Documents/software/python/pyemd/pyemd/emd.cpython-35m-darwin.so
Expected in: dynamic lookup
)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/mortonjt/miniconda2/envs/bio/lib/python3.5/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/Users/mortonjt/miniconda2/envs/bio/lib/python3.5/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/mortonjt/miniconda2/envs/bio/lib/python3.5/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/mortonjt/miniconda2/envs/bio/lib/python3.5/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/mortonjt/miniconda2/envs/bio/lib/python3.5/imp.py", line 244, in load_module
return load_package(name, filename)
File "/Users/mortonjt/miniconda2/envs/bio/lib/python3.5/imp.py", line 216, in load_package
return _load(spec)
File "<frozen importlib._bootstrap>", line 693, in _load
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/Users/mortonjt/Documents/software/python/pyemd/pyemd/__init__.py", line 60, in <module>
from .emd import emd
ImportError: dlopen(/Users/mortonjt/Documents/software/python/pyemd/pyemd/emd.cpython-35m-darwin.so, 2): Symbol not found: __ZNSt8__detail15_List_node_base7_M_hookEPS0_
Referenced from: /Users/mortonjt/Documents/software/python/pyemd/pyemd/emd.cpython-35m-darwin.so
Expected in: dynamic lookup
pip install gives the exact same issue.
I'm also using python3.5.
Do you have any suggestions around this? Thanks!
In my project there is a dependency on textacy, that depends on PyEMD.
It seems I cannot install this package using pip by running pip install -r requirements.txt --no-cache-dir. Pip itself is installed using anaconda, if that's relevant.
The requirements.txt looks as follows:
textacy==0.7.0 --> depends on pyemd >= 0.3, so tries to get 0.5.1(latest as of now)
pandas==0.21
gensim==3.3.0
xlrd==1.1.0
abbreviations==0.1.5
spacy-langdetect==0.1.2
PyPDF2==1.26.0
matplotlib==3.1.0
wordcloud==1.5.0
scattertext==0.0.2.52
I've tried changing the numpy version, as I thought it might have to do with numpy version compatibility. Using the latest version(1.18.1) of numpy does fix this.
Collecting pyemd>=0.3.0 (from textacy==0.7.0->-r requirements.txt (line 1))
Downloading ***/api/pypi/releases_pypi_all/packages/packages/c0/c5/7fea8e7a71cd026b30ed3c40e4c5ea13a173e28f8855da17e25271e8f545/pyemd-0.5.1.tar.gz (91kB)
100% |████████████████████████████████| 92kB 5.0MB/s
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/pyemd.egg-info
writing pip-egg-info/pyemd.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/pyemd.egg-info/dependency_links.txt
writing requirements to pip-egg-info/pyemd.egg-info/requires.txt
writing top-level names to pip-egg-info/pyemd.egg-info/top_level.txt
writing manifest file 'pip-egg-info/pyemd.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-dr1_6j7o/pyemd/setup.py", line 121, in <module>
'Programming Language :: Python :: 3.6',
File "/home/user/.local/lib/python3.6/site-packages/setuptools/__init__.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/home/user/Applications/anaconda3/envs/py36_txt2insights/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/home/user/Applications/anaconda3/envs/py36_txt2insights/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/home/user/Applications/anaconda3/envs/py36_txt2insights/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/home/user/.local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
File "/home/user/.local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
File "/home/user/.local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
File "/home/user/.local/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
File "/home/user/.local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/home/user/.local/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/home/user/Applications/anaconda3/envs/py36_txt2insights/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
cmd_obj.ensure_finalized()
File "/home/user/Applications/anaconda3/envs/py36_txt2insights/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
self.finalize_options()
File "/tmp/pip-install-dr1_6j7o/pyemd/setup.py", line 78, in finalize_options
self.include_dirs.append(numpy.get_include())
AttributeError: module 'numpy' has no attribute 'get_include'
When calculating the optimal flow the algorithm is correct but seems like the EMD is a little off from what I would correctly get with flow_matrix * distance_matrix
Distance matrix:
[[2.2360679775e+00 8.7952156412e+03 3.1622776602e+00]
[8.7964805272e+03 0.0000000000e+00 8.7921069089e+03]
[0.0000000000e+00 8.7964805272e+03 5.0000000000e+00]]
arr1:
[0.5 0.5 0.0]
arr2:
[0.0 0.5 0.5]
emd = pyemd.emd_with_flow(arr1, arr2, distance_matrix)
= (1.57896825463524, [[0.0, 0.0, 0.5], [0.0, 0.5, 0.0], [0.0, 0.0, 0.0]])
The final value should be
np.sum(np.multiply(emd[1], distance_matrix))
= 0.5 * 3.1622776602 = 1.5811388301 (=sqrt(2.5))
but instead it is 1.57896825463524 (err=0.002170575465)
Types (distance, arr1, arr2): {dtype} float64
System: Mac OS X, 10.13.4
PIP pyemd: 0.5.1
Seems like too big of an error to be explained by floating point precision. Also, for floating point precision error the calculated flow should be off as well, but that seems to be accurate.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.