Just a quick update on this:

@DanielLenz: It seems like matplotlib was indeed the issue. The kernel only dies when I import both nmt and matplotlib together. I didn't try the different backends, but os.environ["KMP_DUPLICATE_LIB_OK"] = "True" resolves the issue.

Tagging @lluism and @izzyswafford because we ran into a similar issue a while ago.

I'm still looking forward to comments from other people as well.

from namaster.

What's the jupyter error message, i.e. what is printed in the terminal that runs your jupyter server?

from namaster.

Hey Daniel,

This is what I get in the terminal:

Adapting from protocol version 5.1 (kernel 7b812c0f-23d8-4dd2-9542-d8014eb96ca1) to 5.3 (client). OMP: Error #15: Initializing libomp.dylib, but found libiomp5.dylib already initialized. OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

Is there a quick fix for this?

from namaster.

I haven't experienced this issue myself, but quickly googling the error (e.g. dmlc/xgboost#1715) suggests a couple of different approaches:

• Re-installing pymaster (did you install using conda? I think that's the recommend way nowadays)
• Use the "unsafe, unsupported, undocumented workaround ", i.e. os.environ["KMP_DUPLICATE_LIB_OK"] = "True"
• Don't use notebooks (worthwhile talk by Joel Grus)
• Use a different matplotlib backend (TkAgg can apparently cause issues)
• conda install nomkl
• Wait for others to chime in!

Good luck!

from namaster.

Thanks, Daniel. I did install pymaster using conda, but I'll definitely try your other suggestions and post the updates here soon.

Cheers

from namaster.

I am also running into similar issues and if you import namaster first, the problem doesn't show (at least in my case).

from namaster.

@fjaviersanchez: Thanks for the update and sorry for my late reply. I just tested this and the problem still persists.

The kernel dies regardless of the import order for namaster and matplotlib.

from namaster.

@syasini thanks for trying this workaround and reporting the results here.

from namaster.

Hello all,
I am facing a somewhat similar issue and can't crack it so I was wondering if someone out there would have any tip. I installed Namaste from source and can test the C routines check just fine (make check) but running the python test crashed my python (very unusual) with a segfault.
Below is the terminal output. It suggests something is off in my setup but it usually works well so I am confused. Any suggestion welcome and thanks for making this package public.

===
bash-3.2$ python -m unittest discover -v
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/healpy/fitsfunc.py:351: UserWarning: If you are not specifying the input dtype and using the default np.float64 dtype of read_map(), please consider that it will change in a future version to None as to keep the same dtype of the input file: please explicitly set the dtype if it is important to you.
warnings.warn(
test_bins_flat_alloc (test.test_nmt_bins.TestBinsFsk) ... ok
test_bins_flat_binning (test.test_nmt_bins.TestBinsFsk) ... ok
test_bins_flat_errors (test.test_nmt_bins.TestBinsFsk) ... ok
test_bins_binning (test.test_nmt_bins.TestBinsSph) ... ok
test_bins_binning_f_ell (test.test_nmt_bins.TestBinsSph) ... ok
test_bins_constant (test.test_nmt_bins.TestBinsSph) ... ok
test_bins_edges (test.test_nmt_bins.TestBinsSph) ... ok
test_bins_errors (test.test_nmt_bins.TestBinsSph) ... ok
test_bins_variable (test.test_nmt_bins.TestBinsSph) ... ok
test_workspace_covar_flat_benchmark (test.test_nmt_covar.TestCovarFsk) ... ok
test_workspace_covar_flat_errors (test.test_nmt_covar.TestCovarFsk) ... ok
test_workspace_covar_benchmark (test.test_nmt_covar.TestCovarSph) ... /opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/healpy/fitsfunc.py:351: UserWarning: If you are not specifying the input dtype and using the default np.float64 dtype of read_map(), please consider that it will change in a future version to None as to keep the same dtype of the input file: please explicitly set the dtype if it is important to you.
warnings.warn(
/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/healpy/fitsfunc.py:351: UserWarning: If you are not specifying the input dtype and using the default np.float64 dtype of read_map(), please consider that it will change in a future version to None as to keep the same dtype of the input file: please explicitly set the dtype if it is important to you.
warnings.warn(
Segmentation fault: 11
bash-3.2$

from namaster.

Hi @olivierdore, this looks like conflicting healpix-related libraries (maybe NaMaster is calling one healpix/libsharp version and healpy a different one?) because it breaks the first time it's calling healpy, essentially. I also thought it could be python3.8 problems but I just installed from source using py38 and it seems to work for me. Have you tried using pip install pymaster (this is likely still not going to fix it but who knows?) or if you use conda conda install pymaster -c conda-forge? @damonge, any ideas?

from namaster.

This does not exactly answer @olivierdore's question, but a good workaround solution for the problem would be to use the Docker image from here: https://github.com/simonsobs/PSpipe
I hope this helps.

from namaster.

Thank you @fjaviersanchez and @syasini . I will fight a bit more before installing the Docker which I am sure would work. @fjaviersanchez I have been careful to link directly to my healpix/healpy installation (through MacPorts) and not reinstall it. For some reason, I do need however to reinstall libsharp as using my default installation (through MacPorts) leads to a segfault in the C "make check" (TEST 23/66 nmt:he_sht_car /bin/sh: line 1: 79903 Segmentation fault: 11 ${dir}$tst), whereas linking to a fresh install leads to the segfault in python... Very odd. But maybe libsharp is indeed the culprit...

from namaster.

I think that NaMaster is still reinstalling healpix and libsharp. A hacky option is to copy or create a symlink of libchealpix.a and libsharp.a in the ./_deps/lib subdirectory, where ./ is NaMaster's top directory (where setup.py lives) and also copy (or symlink) the files from the include subdirectories from libsharp and healpix and put them in _deps/include. That way the reinstall part is skipped. Another hacky option is to comment out lines 76 and 77 in setup.py and remove "./_deps/lib/libchealpix.a" from line 91 and see if it still compiles.

from namaster.

Thank you Javier. I tried all the above already but it did not solve my problem. Either I break the C install or I break the python wrappers!

from namaster.

We've never tested the code on python3.8. I doubt that's the issue, but I'll give it a try.

Can I check if you have:

• Removed all tracers of priors installations of libsharp and namaster
• Reinstalled namaster through pip.
  ?

from namaster.

Travis passes on 3.8, so that's not the problem

from namaster.

Hi @damonge . Thanks for the answer. I have libsharp install in my libary as it comes with healpy (all installed through MacPorts) and can not remove it without breaking healpy. Pip (also from MacPorts) does not work for me as it does not find the dependencies (fftw, gsl...) even though they are there (this is true whether I use CC=gcc or clang). That's why I tried to install from the source. I now suspect the issue is that sharp was installed with clang (that's the only option for macports) and I was trying to install namaster with gcc. It does not affect the C build up/check though... so I am retrying to reinstall namaster with clang now but still face some segfault testing the C library (it compiles fine).

from namaster.

Hey all, just wanted to give you an update. After reinstalling from scratch, linking to the dependent libraries by hand (ie explicit sim links) to make sure I can control all the dependencies, and deciding to use only clang (although it should not matter), I can run properly the C routines "make check" but I still get a segfault 11 using the python wrapper in (test_workspace_covar_benchmark (test.test_nmt_covar.TestCovarSph). I am running out of ideas so will give up for now and will move to another machine. Thanks for the suggestions. In the process, I discovered that I could not link properly with the macports installed sharp library (installed for healpix/healpy) and I had to reinstall a new version to get it to work properly. Maybe that is the issue after all but why it would affect the python call of this library and not the C call is beyond me. Thanks.

from namaster.