Strange Cpu time usage with multiple MPI processes on tutorial example thermal_plasma_1d.py about smilei HOT 6 CLOSED

Hi
How many openMP threads do you use ? This behaviour is typical of an oversubscribing of the cores of your CPU. Either because you have too many threads per core and use hyperthreading or simply because you pin your openMP threads on the same core. Make sure your openMP threads are correctly pinned.
When not specified sometimes the maximum number of openMP threads are used by certain system and that leads to bad performances. If you want to run the code without openMP at all and just run mpi checks you can compile the code with the option config=nopenmp.
Let us know how that goes.

from smilei.

I confirm Arnaud's comment.
You'll find below the behaviour observed on a double E5-2670 architecture (fixing OMP_NUM_THREADS to 1).
It has been run with IntelMPI which is smart regarding the MPI process affinity without specifying it.

from smilei.

OK you are right,
For any reason when OMP_NUM_THREADS is not set, the scalability problem i mentioned in this issue
shows up immediately after more than 2 MPI cores. So one has to explicitly set this env. variable and the speed up is clea and linear when using n= 1 or 2 threads but for higher threads number is see no real improvement. May be some other dual MPI - OpenMP effects ?...

from smilei.

As I said in my first message, if you do not set your OMP_NUM_THREADS, sometimes the system uses a stupid value. The SMILEI simulation log tells you how many openMP threads are being used so that you can check this.
You should be able to reproduce an almost linear improvement for openMP as well but it is more difficult to achieve because you have to properly pin your threads to the cores.
The first thing we advise is not to use openMP across different sockets. So make sure you allocate at least one MPI process per socket.
Then you have to allocate at least as many cores to each of your MPI process as you have openMP threads.
It is often good to bind threads to cores.
And finally make sure your threads are properly positioned across your physical cores and not stack on each other on a same core because of hyperthreading.

All of this is completely independent of SMILEI and is just standard good practice of hybrid codes on many core systems. It is difficult to tell you exactly how to do it because it depends on your environment, compiler, Mpi version etc.

from smilei.

thanks a lot for the detailed explanation.

from smilei.

Anytime. Hope this will help you get the most out of SMILEI. Let us know if we can be of further help.

from smilei.