I'm using IPM as a profiler for OpenFoam, but sometimes when I do the ipm_parse phase I get the following error.

sh: -c: line 0: syntax error near unexpected token ('`

this creates a file called
interFoam (deleted)_12_smartini.1540304391.359668.ipm.xml_ipm_2170337.master1'
without having generated the figures.

I can not understand why, by launching the same job sequentially but with a different number of processes, I get or not get the graphs correctly.

In the loaded zip I have two examples one working and one not working.

Archive.zip

In order to make IPM usable with recent versions of OpenMPI, commit ce13c7f introduced the idea of having the IPM Fortran MPI wrappers call the Fortran PMPI interface directly as opposed to type conversion and calls to the C PMPI interface.

One consequence of this is that even in a pure-C MPI code, one may need to link with the MPI Fortran compiler, lest there be undefined references to the Fortran PMPI routines.

Instead, as pointed out by @ggouaillardet in the comment to 5901492, splitting the IPM C and Fortran intercept libraries such that C-only codes need only link with -lipm, while Fortran codes would use -lipmf -lipm would be a nice alternative.

Hi there,

Great to see development of IPM continuing, thanks for such a great tool :)

Version 0.983 could measure how much each rank talked to each other rank, which ended up in the Communication Topology graph. This seems to have disappeared in 2.x, but I see that the old code is still kicking around in there.

Are there any plans to get this working again, please?

Cheers,

Mark

Is there a paper describing IPM?

How to compile MPICH source code itself with IPM profiler

Using a new libtool with intel 2019 compiler, I get failures in make install step because of missing links (which are not created):

libtool: link: (cd ".libs" && rm -f "libipmf.so.0" && ln -s "libipmf.so.0.0.0" "libipmf.so.0")
libtool: link: (cd ".libs" && rm -f "libipmf.so" && ln -s "libipmf.so.0.0.0" "libipmf.so")
libtool: link: ar cru .libs/libipmf.a  libipmf_la-mpi_fortran.o libipmf_la-GEN.wrapper_mpif.o
libtool: link: ranlib .libs/libipmf.a
libtool: link: ( cd ".libs" && rm -f "libipmf.la" && ln -s "../libipmf.la" "libipmf.la" )
 /usr/bin/mkdir -p '/usr/common/software/ipm/2.0.5/intel/lib'
 /bin/sh ../libtool   --mode=install /usr/bin/install -c   libipm.la libipmf.la '/usr/common/software/ipm/2.0.5/intel/lib'
libtool: install: /usr/bin/install -c .libs/libipm.so.0.0.0 /usr/common/software/ipm/2.0.5/intel/lib/libipm.so.0.0.0
/usr/bin/install: cannot stat '.libs/libipm.so.0.0.0': No such file or directory
make[2]: *** [Makefile:761: install-libLTLIBRARIES] Error 1
make[2]: Leaving directory '/global/project/projectdirs/swowner/tkurth/modules/ipm/cori_build/src'
make[1]: *** [Makefile:1090: install-am] Error 2
make[1]: Leaving directory '/global/project/projectdirs/swowner/tkurth/modules/ipm/cori_build/src'
make: *** [Makefile:421: install-recursive] Error 1```

I'm trying to make OpenFoam profiling with IPM. But I found a mistake when I use ploticus.

This is what I get

[smartini@login2 damBreak]$ /home/smartini/ipm_2.0.3/bin/ipm_parse -html smartini.1504091236.699412.ipm.xml
data_acquire = 0 sec
data_workup = 0 sec
mpi_pie = 0 sec
task_data = 0 sec
load_bal = 0 sec
time_stack = 0 sec
mpi_stack = 0 sec
mpi_buff = 0 sec
switch+mem = 0 sec
topo_tables = 0 sec
topo_data = 0 sec
topo_time = 0 sec
html_all = 0 sec
html_regions = 0 sec
pl proc areadef: error 100: y range is invalid .. likely culprit is yautorange or yrange
pl proc areadef: error 10: cannot set up plotting area .. likely culprits: bad xrange or yrange, or bad area rectangle
html_nonregion = 0 sec

I attached here my xml file

smartini.1504091236.699412.ipm.xml.zip

I tried to do the tests I found in ipm and they did not give me this problem.

Kind Regard
Simone

Dear admin,

I have the error during linking the IPM project:
/bin/sh ../libtool --tag=FC --mode=link mpif90 -mcpu=elbrus-4c -fuse-glibc -mcpu=elbrus-4c -fuse-glibc -L/usr/lib64 -o libipmf.la -rpath /home/ordi/inst/IPM/lib libipmf_la-mpi_fortran.lo libipmf_la-GEN.wrapper_mpif.lo
libtool: link: mpif90 -shared .libs/libipmf_la-mpi_fortran.o .libs/libipmf_la-GEN.wrapper_mpif.o -L/usr/lib64 -L/opt/mpich2-1.1.1p1/lib -L/opt/mcst/lcc-home/1.19.21/e2k-4c-linux/lib/ptr64 -lmpichf90 -lmpich -lopa -lpthread -lrt /usr/lib/libfcc.so -lm -lmvec -lgcc -lgcc_eh -lc -mcpu=elbrus-4c -mcpu=elbrus-4c -soname libipmf.so.0 -o .libs/libipmf.so.0.0.0
lcc: error: file or directory "libipmf.so.0" is corrupted and unreadable

./configure --prefix=/home/ordi/inst/IPM/ FC=gfortran --with-gnu-ld

I have mpich2: 1.1.1p1 and gcc version 4.4.0 compatible.

What am I doing wrong? I've tried 2.0.3, 2.04 and 2.0.5 releases.
Thank you!

Best regards,
ordi

my env:

$spack find --loaded
==> 14 loaded packages
-- linux-centos7-skylake_avx512 / [email protected] ---------------------
[email protected]    [email protected]  [email protected]        [email protected]  [email protected]  [email protected]
[email protected]  [email protected]          [email protected]  [email protected]      [email protected]  [email protected]

-- linux-centos7-skylake_avx512 / [email protected] ---------------------
[email protected]  [email protected]

 $module list

Currently Loaded Modules:
  1) NiaEnv/2019b   2) gcc/8.3.0

configure:

./configure --enable-shared  --enable-parser --enable-posixio --with-papi=/xxxxxxxxxxxxxx/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/papi-6.0.0.1-btlnyicokbpyo222py5daqsw5kwvmjrs LDFLAGS=-L/xxxxxxxxxxxxxxx/mxml CFLAGS=-fPIC CPPFLAGS=-I/xxxxxxxxxxxxxx/mxml FC=gfortran

1.The generated file src/Makefile FC = is empty,I add it manually.
2.Commands in src/Makefile target libipm.la and libipmf.la shoud change from rpath to -Wl,rpath=
Then it can be compiled and libipm.a and libipmf.a generated in src/.libs. but libipm.so and libipmf.so not be generated.

I compile libipm.so and libipmf.so manually using the following command.
libipm.so

mpicc -shared  -DPIC  .libs/libipm_la-calltable.o .libs/libipm_la-hashkey.o .libs/libipm_la-hashtable.o .libs/libipm_la-ipm_core.o .libs/libipm_la-ipm_env.o .libs/libipm_la-ipm_time.o .libs/libipm_la-ipm_modules.o .libs/libipm_la-jobdata.o .libs/libipm_la-md5.o .libs/libipm_la-memusage.o .libs/libipm_la-perfdata.o .libs/libipm_la-regstack.o .libs/libipm_la-report.o .libs/libipm_la-report_banner.o .libs/libipm_la-report_xml.o .libs/libipm_la-ipm_introspect.o .libs/libipm_la-mod_mpi.o .libs/libipm_la-mpi_init.o .libs/libipm_la-mpi_finalize.o .libs/libipm_la-mpi_pcontrol.o .libs/libipm_la-machtopo.o .libs/libipm_la-mod_posixio.o .libs/libipm_la-mod_papi.o .libs/libipm_la-GEN.wrapper_mpi.o .libs/libipm_la-GEN.calltable_mpi.o .libs/libipm_la-GEN.wrapper_posixio.o .libs/libipm_la-GEN.fake_posixio.o .libs/libipm_la-GEN.calltable_posixio.o   -L/xxxxxxxxxxxxx/mxml -L/xxxxxxxxxxxxx/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/papi-6.0.0.1-btlnyicokbpyo222py5daqsw5kwvmjrs/lib -lmxml -lpapi -lc  -Wl,-rpath=/xxxxxxxxxxxxx/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/papi-6.0.0.1-btlnyicokbpyo222py5daqsw5kwvmjrs/lib   -Wl,-soname=libipm.so.0 -o .libs/libipm.so.0.0.0

libipmf.so

mpifort -shared  -DPIC .libs/libipmf_la-mpi_fortran.o .libs/libipmf_la-GEN.wrapper_mpif.o -L/scratch/l/lcl_uotiscscc/lcl_uotiscsccs1035/mxml -L/xxxxxxxxxxxxxxx/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/papi-6.0.0.1-btlnyicokbpyo222py5daqsw5kwvmjrs/lib -lmxml -lpapi -lc  -Wl,-rpath=/xxxxxxxxxxxxxxxxxx/spack/opt/spack/linux-centos7-skylake_avx512/gcc-9.3.0/papi-6.0.0.1-btlnyicokbpyo222py5daqsw5kwvmjrs/lib   -Wl,-soname=libipmf.so.0 -o .libs/libipmf.so.0.0.0

I test with my [email protected] and it works well.

Is there any way to correct the configure script to fix the wrong makefile?
Why shared library not being generated?

I use IPM-2.0.6 to stat MPI communication behavior while Communication Topology is missing when ipm_parse to html ?

Hi there! I need your help, please :)

I begin by installing Ubuntu 16.04 in a Virtual Machine (for testing purposes), with 1 processor and 2 cores

Then install basic stuff (gcc, g++, and fortran compilers):

sudo apt-get update
sudo apt-get install build-essential autoconf automake git
sudo apt-get install gcc g++ gfortran fort77

Then install OpenMPI:

sudo apt-get update
sudo apt-get install openmpi-bin openmpi-common openssh-client openssh-server libopenmpi-dev

Then install IPM:

cd  #go home
git clone https://github.com/nerscadmin/IPM.git
cd IPM

sudo -s #become root 
./bootstrap.sh
apt-get install libmxml1 libmxml-dev libunwind8

./configure --enable-coll-details --enable-parser

Produces:
_################################################
IPM configuration:
MPI profiling enabled : yes
POSIX-I/O profiling enabled : no
PAPI enabled : no
CFLAGS : -g -O2 -DHAVE_DYNLOAD -DIPM_COLLECTIVE_DETAILS=1 -DOS_LINUX
LDFLAGS :
LIBS : -lmxml
MPI_STATUS_COUNT : ucount
Fortran underscore : -funderscore_post
Building IPM Parser : yes

make
make install
exit #to exit from root

cd ~/IPM/test
make

cd ~/IPM/utils
./make_mxml

And then, when testing:

export LD_PRELOAD=/usr/local/lib/libipm.so
export IPM_REPORT=full
export IPM_REPORT_MEM=yes
export IPM_LOG=full 
unset IPM_HPM
cd ~/IPM/test/test.allgather
mpirun -np 2 allgather

It generates a file jcarrete.1462829706.ipm.xml (around 7KB, file attached)
Then, I install ploticus:

sudo apt-get install ploticus

And parse the xml file:

export IPM_KEYFILE=~/IPM/etc/ipm_key_mpi
ipm_parse -html jcarrete.1462829706.ipm.xml

The result is that it generates output to the stdout, but does not generate any directory.
If I execute:

ipm_parse -html file jcarrete.1462829706.ipm.xml -o output.html

Then, it generates an html file with almost no information, no navigation options and fails to load these files: https://dl.dropboxusercontent.com/u/408013/protovis.js and http://dl.dropbox.com/u/408013/style.css (output.html attached)

Output files: outputs.zip

Am I doing something wrong? Please help me :)

Hello, dear Scott!

It's me again :(
Now I have problem while linking IPM-2.0.5 (or using PRELOAD) with Intel MPI fortran (mpiifort).
Error when dynamic linking looks like:
/bin/sh: symbol lookup error: /home/ordi/inst/IPM1/lib/libipmf.so: undefined symbol: ipm_in_fortran_pmpi
static - during compilation:
_/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Rsend' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toipm_in_fortran_pmpi'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Gatherv' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Bsend'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Bsend_init' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Ssend'
etc...

Addition information:
Application - NAS Parallel Benchmark, NPB3.3-MPI (test FT, for example)

## Dynamic link
./configure --prefix=/home/ordi/inst/IPM1 --with-papi=/usr --enable-share MPICC=mpiicc MPIF77=mpif77 MPIFC=mpiifor
IPM configuration:
MPI profiling enabled : yes
POSIX-I/O profiling enabled : no
PAPI enabled : yes
CFLAGS : -DHAVE_DYNLOAD -I/usr/include -DOS_LINUX
LDFLAGS : -L/usr/lib -Wl,-rpath=/usr/lib
LIBS : -lpapi
MPI_STATUS_COUNT : count_lo
Fortran underscore : -funderscore_post
Building IPM Parser : no

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ordi/inst/IPM1/lib

LD_PRELOAD=/home/ordi/inst/IPM1/lib/libipmf.so mpirun -np 4 ./ft.B.4
/bin/sh: symbol lookup error: /home/ordi/inst/IPM1/lib/libipmf.so: undefined symbol: ipm_in_fortran_pmpi

## STATIC LINK:
Configuration options:
./configure --prefix=/home/ordi/inst/IPM1 --with-papi=/usr --enable-static MPICC=mpiicc MPIF77=mpif77 MPIFC=mpiifort
IPM configuration:
MPI profiling enabled : yes
POSIX-I/O profiling enabled : no
PAPI enabled : yes
CFLAGS : -DHAVE_DYNLOAD -I/usr/include -DOS_LINUX
LDFLAGS : -L/usr/lib -Wl,-rpath=/usr/lib
LIBS : -lpapi
MPI_STATUS_COUNT : count_lo
Fortran underscore : -funderscore_post
Building IPM Parser : no

Application Makefile:
MPIFC = mpiifort
FMPI_LIB = -L/opt/intel/impi/5.0.1.035/lib64/ -lmpi
FMPI_INC = -I/opt/intel/impi/5.0.1.035/include64/ -I/home/ordi/inst/IPM1/include
FFLAGS = -O3
FLINKFLAGS = -O3 -L/home/ordi/inst/IPM1/lib -lipmf

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH: /home/ordi/inst/IPM1/lib
Error during compilation:
make ft CLASS=B NPROCS=4

= NAS Parallel Benchmarks 3.3 =
= MPI/F77/C =

cd FT; make NPROCS=4 CLASS=B
make[1]: Entering directory /home/ordi/src/NPB3.3.1/NPB3.3-MPI/FT' make[2]: Entering directory/home/ordi/src/NPB3.3.1/NPB3.3-MPI/sys'
cc -g -o setparams setparams.c
make[2]: Leaving directory /home/ordi/src/NPB3.3.1/NPB3.3-MPI/sys' ../sys/setparams ft 4 B mpiifort -c -I/opt/intel/impi/5.0.1.035/include64/ -I/home/ordi/inst/IPM1/include -O3 ft.f cd ../common; mpiifort -c -I/opt/intel/impi/5.0.1.035/include64/ -I/home/ordi/inst/IPM1/include -O3 randi8.f cd ../common; mpiifort -c -I/opt/intel/impi/5.0.1.035/include64/ -I/home/ordi/inst/IPM1/include -O3 print_results.f cd ../common; mpiifort -c -I/opt/intel/impi/5.0.1.035/include64/ -I/home/ordi/inst/IPM1/include -O3 timers.f mpiifort -O3 -L/home/ordi/inst/IPM1/lib -lipmf -o ../bin/ft.B.4 ft.o ../common/randi8.o ../common/print_results.o ../common/timers.o -L/opt/intel/impi/5.0.1.035/lib64/ -lmpi /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Rsend'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to ipm_in_fortran_pmpi' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Gatherv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Bsend' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Bsend_init'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Ssend' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Irecv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Send_init' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Buffer_detach'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Recv_init' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Reduce'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Barrier' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Ssend_init'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Allgather' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Recv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Iprobe' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Sendrecv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Issend' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Alltoallv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Ibsend' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toipm_state'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Alltoall' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Comm_compare'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Irsend' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Isend'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Scatter' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Reduce_scatter'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Send' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Comm_rank'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Buffer_attach' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Start'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Gather' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Scatterv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Comm_free' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Sendrecv_replace'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Allreduce' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Comm_split'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Comm_group' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Rsend_init'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Comm_create' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Scan'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Bcast' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Allgatherv'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Probe' /home/ordi/inst/IPM1/lib/libipmf.so: undefined reference toIPM_MPI_Comm_size'
/home/ordi/inst/IPM1/lib/libipmf.so: undefined reference to IPM_MPI_Comm_dup' make[1]: Leaving directory/home/ordi/src/NPB3.3.1/NPB3.3-MPI/FT'

PS: If I link application with -lipm flag instead of -lipmf - then application linked successfully. But there are no IPM effect during running - no xml, no statistic, nothing.

I'm trying to profile different sections of a solver in OpenFOAM, on small cases this works quite well.
However for large cases I need tens of runs on the cluster to complete my case.

I would (ideally) like to put a call like:

ipm_output();

at the end of my time-loop (every time-step can take hours to complete) such that I have a record of the time spent in my defined sections.

Alternatively, I would have to implement some sort of interrupt signal for OpenFOAM to exit the program once I near the end of my time-reservation which seems to be difficult

I tested IPM master with both openmpi and mpich, and wrote a simple fortran test program similar to tests/test.alltoall
IPM reports twice the number of calls to MPI_Alltoall (!)
the root cause is fortran pmpi_alltoall_ calls IPM MPI_Alltoall instead of the MPI PMPI_Alltoall symbol, and IPM does nothing to avoid double accounting
fwiw, I asked the openmpi folks to comment at http://www.open-mpi.org/community/lists/devel/2015/08/17842.php since that issue can be solved within openmpi (I cannot speak for other MPI flavors, many of them being mpich derivatives)
imho, avoiding double accounting is pretty trivial to avoid in IPM unless MPI_THREAD_MULTIPLE is used.

I cannot open http://ipm-hpc.org/. Does anyone have the same issue?

The old site does work, though: http://ipm-hpc.sourceforge.net

Dear admin (Scott),

I have IPM configured at Haswell CPU with dynamic preload.
Final picture contains empty Communication Event Statistics and communication balance picture, while there where 56 MPI tasks with MPI_reduce and Barrier operations.

Could you please tell me, what am I doing wrong?

Thank you for your time!

Please, find attached final html, if needed.

Other details:

Configuration options:
./configure --prefix=/home/ordi/inst/IPM CC=icc CXX=icpc --with-papi=/usr --with-libunwind=/usr CXX=icpc MPICC=mpiicc MPIFC=mpiifort --enable-shared LDFLAGS=-lpthread --enable-coll-details
Configuration:

IPM configuration:
MPI profiling enabled : yes
POSIX-I/O profiling enabled : no
PAPI enabled : yes
use libunwind : yes
CFLAGS : -DHAVE_DYNLOAD -DIPM_COLLECTIVE_DETAILS=1 -I/usr/include -I/usr/include -DOS_LINUX
LDFLAGS : -lpthread -L/usr/lib -Wl,-rpath=/usr/lib -L/usr/lib
LIBS : -lpapi -lunwind
MPI_STATUS_COUNT : count_lo
Fortran underscore :
Building IPM Parser : no

I set environment vars:
export IPM_HPM=PAPI_TOT_INS,PAPI_L1_DCM,PAPI_TOT_CYC
export IPM_REPORT=full

run program with preload:
LD_PRELOAD=/home/ordi/inst/IPM/lib/libipm.so mpirun -np 56 ./wemig2dmpi
and set
export IPM_KEYFILE=/home/ordi/inst/IPM/etc/ipm_key_mpi
result.tar.gz
and parse:
/home/ordi/inst/IPM/bin/ipm_parse -html wemig56.xml

Best regards,
ordi

Dear Admin,
I think I may have found a problem when telling IPM to profile long PAPI event names.

For example, profiling a valid native PAPI event through IPM_HPM like
export IPM_HPM=DTLB_LOAD_MISSES:MISS_CAUSES_A_WALK
will make IPM crash.

*** buffer overflow detected ***: bin/mpi terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x77725)[0x7f5317326725]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f53173c789c]
/lib/x86_64-linux-gnu/libc.so.6(+0x1168a0)[0x7f53173c58a0]
/lib/x86_64-linux-gnu/libc.so.6(+0x115bf2)[0x7f53173c4bf2]
/home/jcarrete/installations/IPM/lib/libipm.so(ipm_check_env+0x2bc)[0x7f5317b7579c]
/home/jcarrete/installations/IPM/lib/libipm.so(ipm_get_env+0x231)[0x7f5317b75c31]
/home/jcarrete/installations/IPM/lib/libipm.so(ipm_init+0xac)[0x7f5317b74e3c]
/home/jcarrete/installations/IPM/lib/libipm.so(MPI_Init+0x23)[0x7f5317b7c673]
bin/mpi[0x400c56]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f53172cf830]
bin/mpi[0x400fb9]
======= Memory map: ========

I have modified the size for MAXSIZE_PAPI_EVTNAME in /include/ipm_sizes.h to acommodate for bigger event names, and it seems to work, but I don't know the implications of this change.

Javi Carretero

[root@localhost em_real]# mpirun --allow-run-as-root --hostfile myhost -x LD_PRELOAD -np 4 ./hello_c
Hello, world, I am 0 of 4, (Open MPI v3.1.0, package: Open MPI [email protected] Distribution, ident: 3.1.0, repo rev: v3.1.0, May 07, 2018, 121)
Hello, world, I am 2 of 4, (Open MPI v3.1.0, package: Open MPI root@arm-node01 Distribution, ident: 3.1.0, repo rev: v3.1.0, May 07, 2018, 110)
Hello, world, I am 1 of 4, (Open MPI v3.1.0, package: Open MPI [email protected] Distribution, ident: 3.1.0, repo rev: v3.1.0, May 07, 2018, 121)
Hello, world, I am 3 of 4, (Open MPI v3.1.0, package: Open MPI root@arm-node01 Distribution, ident: 3.1.0, repo rev: v3.1.0, May 07, 2018, 110)
^CIPM 0: ERROR ipm_finalize() called with ipm_state=4
IPM 1: ERROR ipm_finalize() called with ipm_state=4
IPM 3: ERROR ipm_finalize() called with ipm_state=4
IPM 2: ERROR ipm_finalize() called with ipm_state=4

Hello, when I run the MPI program on a machine (mpirun --allow-run-as-root -x LD_PRELOAD -np 4 ./hello_c), IPM can calculate OpenMPI performance very well.

But when I run IPM through two machine clusters (mpirun --allow-run-as-root --hostfile myhost -x LD_PRELOAD -np 4 ./hello_c), the program stops. When I manually enter CRTL+C, It will be prompted with an IPM error.

what is this question? Can IPM test MPI cluster performance? Thank you

If PAPI isn't available (e.g. in VMs), apart from the error messages at startup, you get a SEGV writing the report at finalization. I fixed that with this change.

--- IPM-2.0.6/src/report_xml.c~	2016-07-17 00:15:01.000000000 +0100
+++ IPM-2.0.6/src/report_xml.c	2019-04-12 15:37:22.087631803 +0100
@@ -293,7 +293,7 @@
 
   nmod=0;
   for( i=0; i<MAXNUM_MODULES; i++ ) {
-    if( !(modules[i].name) || !(modules[i].xml) ) 
+    if( (modules[i].state == STATE_ERROR) || !(modules[i].name) || !(modules[i].xml) ) 
       continue;
 
     nmod++;
@@ -304,7 +304,7 @@
   /* print the contribution to the task-wide <modules> entry
      or the region-specific <modules> entry for each module */
   for( i=0; i<MAXNUM_MODULES; i++ ) {
-    if( !(modules[i].name) || !(modules[i].xml) ) 
+    if( (modules[i].state == STATE_ERROR) || !(modules[i].name) || !(modules[i].xml) ) 
       continue;
 
     res+=modules[i].xml(&(modules[i]), ptr, reg); 

The default version of Cray's PAPI module has incremented from 5.5.1.4 to 5.6.0.3. This causes applications linked with IPM to crash with this error at run time:

user@nid12345> srun -n 2 ./a.out
IPM  0: ERROR PAPI initialization error (-1)
IPM  0: ERROR Error initializing module 7 (PAPI), error 1
IPM  0: ERROR PAPI initialization error (-1)
IPM  0: ERROR Error initializing module 7 (PAPI), error 1
Hello world from processor nid12345, rank 1 out of 2 processors
Hello world from processor nid12345, rank 0 out of 2 processors
srun: error: nid12345: task 0: Segmentation fault (core dumped)

Swapping the default PAPI module for an older one (e.g., 5.5.1.4) resolves this error.

Built with (OpenMPI 4.1.0a1 and gcc 9.3.0) :
./configure MPICC=mpicc MPIFC=mpif90 --prefix=/install

I get:

libtool: link: mpicc -shared  -DPIC  .libs/libipm_la-calltable.o .libs/libipm_la-hashkey.o .libs/libipm_la-hashtable.o .libs/libipm_la-ipm_core.o .libs/libipm_la-ipm_env.o .libs/libipm_la-ipm_time.o .libs/libipm_la-ipm_modules.o .libs/libipm_la-jobdata.o .libs/libipm_la-md5.o .libs/libipm_la-memusage.o .libs/libipm_la-perfdata.o .libs/libipm_la-regstack.o .libs/libipm_la-report.o .libs/libipm_la-report_banner.o .libs/libipm_la-report_xml.o .libs/libipm_la-ipm_introspect.o .libs/libipm_la-mod_mpi.o .libs/libipm_la-mpi_init.o .libs/libipm_la-mpi_finalize.o .libs/libipm_la-mpi_pcontrol.o .libs/libipm_la-machtopo.o .libs/libipm_la-GEN.wrapper_mpi.o .libs/libipm_la-GEN.calltable_mpi.o   -lc    -soname libipm.so.0 -o .libs/libipm.so.0.0.0
gcc: error: libipm.so.0: No such file or directory
gcc: error: unrecognized command line option '-soname'

Any ideas? I have tried to add -Wl but it does not help.

After a configure step with --enable-posixio and then a make step, we
get the following build error

GEN.wrapper_posixio.c:36:30: error: expected ')' before 'char'
make[2]: *** [libipm_la-GEN.wrapper_posixio.lo] Error 1

Line 36 in GEN.wrapper_posixio.c is clearly nonsense:
FILE* __real_fopen(MPI3CONST char *path, MPI3CONST char *mode);

MPI3CONST should not appear anywhere in the source file because the
POSIX I/O wrappers have nothing to do with MPI. We can fix the problem
by preventing the MPI3CONST substitution in the ipm_key_posixio
template source file. Apply the patch below to the top level IPM
directory:

$ cd IPM/
$ patch -b -p1 < /path/to/ipm-16c494310b_fix_mpi3_subst.diff

$ cat ipm-16c494310b_fix_mpi3_subst.diff

diff -uNr IPM-orig/bin/make_wrappers IPM/bin/make_wrappers
--- IPM-orig/bin/make_wrappers  2014-07-28 14:06:40.000047718 -0700
+++ IPM/bin/make_wrappers   2014-07-28 14:08:02.545538825 -0700
@@ -118,7 +118,11 @@
  $call{$id}{car} =~ s/\(//;
  $call{$id}{car} =~ s/\)//;

- $call{$id}{cai} =~ s/const/MPI3CONST/g;
+ # The I/O functions in ipm_key_posixio, e.g. fwrite, have nothing to
+ # do with MPI and so we avoid the MPI3CONST substitution.
+ if (index("$IPM_KEYFILE", "ipm_key_posixio") == -1) {
+   $call{$id}{cai} =~ s/const/MPI3CONST/g;
+ }

 #
 # cufft

Full steps to reproduce the error:

$ git clone https://github.com/nerscadmin/IPM.git
$ cd IPM/
$ autoreconf -f -i
$ ./configure --disable-mpi --enable-posixio
$ make

/bin/sh ../libtool --tag=CC --mode=compile mpicc -DHAVE_CONFIG_H -I. -I.. -I../include -DMPI_STATUS_COUNT=_ucount -DHAVE_POSIXIO -g -O2 -DHAVE_DYNLOAD -DOS_LINUX -c -o libipm_la-mod_posixio.lo test -f 'mod_posixio.c' || echo './'mod_posixio.c
libtool: compile: mpicc -DHAVE_CONFIG_H -I. -I.. -I../include -DMPI_STATUS_COUNT=_ucount -DHAVE_POSIXIO -g -O2 -DHAVE_DYNLOAD -DOS_LINUX -c mod_posixio.c -fPIC -DPIC -o .libs/libipm_la-mod_posixio.o
libtool: compile: mpicc -DHAVE_CONFIG_H -I. -I.. -I../include -DMPI_STATUS_COUNT=_ucount -DHAVE_POSIXIO -g -O2 -DHAVE_DYNLOAD -DOS_LINUX -c mod_posixio.c -o libipm_la-mod_posixio.o >/dev/null 2>&1
/bin/sh ../libtool --tag=CC --mode=compile mpicc -DHAVE_CONFIG_H -I. -I.. -I../include -DMPI_STATUS_COUNT=_ucount -DHAVE_POSIXIO -g -O2 -DHAVE_DYNLOAD -DOS_LINUX -c -o libipm_la-GEN.wrapper_posixio.lo test -f 'GEN.wrapper_posixio.c' || echo './'GEN.wrapper_posixio.c
libtool: compile: mpicc -DHAVE_CONFIG_H -I. -I.. -I../include -DMPI_STATUS_COUNT=_ucount -DHAVE_POSIXIO -g -O2 -DHAVE_DYNLOAD -DOS_LINUX -c GEN.wrapper_posixio.c -fPIC -DPIC -o .libs/libipm_la-GEN.wrapper_posixio.o
GEN.wrapper_posixio.c:36:30: error: expected ')' before 'char'
GEN.wrapper_posixio.c:57:23: error: expected ')' before 'char'
GEN.wrapper_posixio.c:175:29: error: unknown type name 'MPI3CONST'
GEN.wrapper_posixio.c:196:22: error: unknown type name 'MPI3CONST'
GEN.wrapper_posixio.c:314:32: error: expected ')' before 'char'
GEN.wrapper_posixio.c:335:25: error: expected ')' before 'char'
GEN.wrapper_posixio.c:870:32: error: expected ')' before 'void'
GEN.wrapper_posixio.c:891:25: error: expected ')' before 'void'
GEN.wrapper_posixio.c:1565:34: error: unknown type name 'MPI3CONST'
GEN.wrapper_posixio.c:1586:27: error: unknown type name 'MPI3CONST'
GEN.wrapper_posixio.c:2260:30: error: unknown type name 'MPI3CONST'
GEN.wrapper_posixio.c:2281:23: error: unknown type name 'MPI3CONST'
GEN.wrapper_posixio.c:2399:27: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2420:20: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2538:29: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2559:22: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2677:28: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2698:21: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2955:31: error: expected ')' before 'char'
GEN.wrapper_posixio.c:2976:24: error: expected ')' before 'char'
GEN.wrapper_posixio.c:3233:33: error: expected ')' before 'char'
GEN.wrapper_posixio.c:3254:26: error: expected ')' before 'char'
make[2]: *** [libipm_la-GEN.wrapper_posixio.lo] Error 1

When IPM_LOG=full, IPM will record the source and destination of each message.
The source rank is relative to MPI_COMM_WORLD.
The destination rank is relative to the communicator used to send the message.
Any topology maps based on this data will be incorrect.
For example, the spyplot for GTC should be a banded diagonal matrix, but instead, all of the P2P messages appear in the first block column.

Unfortunately, I don't know of a simple way to fix this.
It would require mapping the dest rank on a subcommunicator to the rank of the same process on COMM_WORLD. I'm not aware of any user-level functions in MPI that would do this.

It would be nice if we could transform/extend IPM to be a darshan module.
The guideline for darshan module developers is here:

http://www.mcs.anl.gov/research/projects/darshan/docs/darshan-modularization.html

With IPM being a darshan module, we can interact with it in a clean and safe way. That way, we could also deploy IPM on Cori Phase 2 once it arrives (which means, that all codes are automatically profiled unless the user opts-out).

To get an impression on how a darshan module looks like, clone https://xgitlab.cels.anl.gov/darshan/darshan.git

and look into the files under darshan-runtime/lib/

Dear Admin,

I have a problem in use PAPI. The point to point data flow data is empty , but in fact my program has mpi point-to-point communication

Dear Scott,
Could you, please, be so kind to help me with cross-compilation of IPM to host Intel Xeon Phi 7120?

./configure --prefix=/home/user14/inst/IPM1 --build=MIC --host=MIC --enable-shared CXXFLAGS='-mmic' CPPFLAGS='-mmic' CXX=icpc CC=icc MPICC=mpiicc MPIF77=mpif77 MPIFC=mpiifort FC=ifort FCFLAGS='-mmic' CPP=icc
checking build system type... Invalid configuration MIC': machineMIC' not recognized
configure: error: /bin/sh ./config.sub MIC failed

thank you...)

Hi all,
When I finish running the application, IPM reports an error and cannot generate an XML file. What should I do?

IPM version: 2.06
Environment settings:
export IPM=/opt/ipm/lib/libipm.so
export IPM_LOGDIR=./${RANDOM}
export IPM_LOG=terse
export IPM_REPORT=full

My run application command:
LD_PRELOAD=$IPM mpirun -n xxxx -x LD_PRELOAD xxxxxx

ipm_parse -html ******489.1684323464.948520.ipm.xml
ERROR non-MPI hent (icall = read) 1.9073e-06 1.9073e-06 1.9073e-06
ERROR non-MPI hent (icall = write) 5.0068e-06 5.0068e-06 5.0068e-06
ERROR non-MPI hent (icall = read) 1.1921e-06 1.1921e-06 1.1921e-06
ERROR non-MPI hent (icall = close) 2.1458e-06 2.1458e-06 2.1458e-06
ERROR non-MPI hent (icall = open) 7.1526e-06 7.1526e-06 7.1526e-06
ERROR non-MPI hent (icall = read) 9.5367e-07 0.0000e+00 9.5367e-07
ERROR non-MPI hent (icall = read) 6.1989e-06 2.1458e-06 4.0531e-06
ERROR non-MPI hent (icall = write) 8.0209e-03 8.0209e-03 8.0209e-03
ERROR non-MPI hent (icall = read) 9.5367e-07 9.5367e-07 9.5367e-07
ERROR non-MPI hent (icall = read) 2.8610e-06 0.0000e+00 9.5367e-07

data_acquire = 0 sec

data_workup = 0 sec

mpi_pie = 0 sec

task_data = 0 sec

load_bal = 1 sec

time_stack = 0 sec

mpi_stack = 0 sec

mpi_buff = 0 sec

switch+mem = 1 sec

topo_tables = 0 sec

topo_data = 0 sec

topo_time = 0 sec

html_all = 2 sec

html_regions = 0 sec

html_nonregion = 0 sec

Dear Admin,

I guess that PAPI event multiplexing is not a very used IPM feature and that it might be experimental.
I have been toying with it and have enabled it by uncommenting /* #define USE_PAPI_MULTIPLEXING */ and by setting MAXNUM_PAPI_EVENTS to 48, MAXNUM_PAPI_COUNTERS to 32, MAXSIZE_PAPI_EVTNAME to 45 and MAXSIZE_ENVKEY to 2048.

When I try to profile a simple test MPI matrix multiplication program (https://github.com/mperlet/matrix_multiplication) having enabled IPM_HPM=IPM_HPM=UOPS_EXECUTED_PORT:PORT_0,PAPI_TOT_CYC,DTLB_LOAD_MISSES:MISS_CAUSES_A_WALK,AVX:ALL,MEM_UOPS_RETIRED:ALL_LOADS,UOPS_EXECUTED:CORE,PAGE-FAULTS:u=0,PAPI_L3_TCM (some randomly selected events) I was getting negative numbers for MEM_UOPS_RETIRED:ALL_LOADS and UOPS_EXECUTED:CORE.

I traced to problem to be occurring in rv = PAPI_set_multiplex(papi_evtset[comp].evtset); inside ipm_papi_startfrom mod_papi.c. I then modified the code embedded in the USE_PAPI_MULTIPLEXING define (for the imp_papi_startfunction) to be like:

#ifdef USE_PAPI_MULTIPLEXING
      rv = PAPI_assign_eventset_component(papi_evtset[comp].evtset, comp);
      if (rv != PAPI_OK) {
    IPMDBG("PAPI: [comp %d] Error calling assign_eventset_component\n", comp);
      }

      rv = PAPI_set_multiplex(papi_evtset[comp].evtset);
      if( rv!= PAPI_OK )    {
    IPMDBG("PAPI: [comp %d] Error calling set_multiplex\n", comp);
      }
#endif

And it seems to be working and returning presumably "correct" values. Does it look good to you? Have you faced similar results previously?

Thank you very much for your help and for this great tool.

Javi Carretero

Profiling an application working on 1024 processes with IPM 2.0.6 we get ~5x overhead.
Analysis showed that ~50% of time application spent in PMPI_Group_compare called from PMPI_Comm_compare().
This application calls a lot of MPI_Isend() for a communicator created by MPI_Cart_create(). Despite the fact that new communicator has the same size and the same process placement as MPI_COMM_WORLD, MPI_Comm_compare() doesn't return MPI_IDENT consuming a lot of computation power for comparison (might be algorithm of PMPI_Group_compare() is not optimal).
Since we still need to call PMPI_Group_translate_ranks() we could compare a communicator with MPI_COMM_WORLD. Something like this (mod_mpi.h):

• PMPI_Comm_compare(MPI_COMM_WORLD, comm_, &comm_cmp_); \
  

• if (comm_cmp_ == MPI_IDENT || rank_in_ == MPI_ANY_SOURCE) { \
  

• if (comm_ == MPI_COMM_WORLD || rank_in_ == MPI_ANY_SOURCE) { \

This modification significantly reduces overhead but it's still huge:
wallclock time with IPM 2.0.2 - 2470s
wallclock time with IPM 2.0.6 - 3400s (with modification above)
Regards!
---Dmitry

The URL/link to Protovis and the CSS stylesheet in 'utils/write_html.cc' is broken. This tool gives little meaning as it is now.

The following call to MPI_Allreduce causes IPM to crash:
call MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, hostnames(1), host_len, MPI_BYTE, MPI_COMM_WORLD, mpierr)

with this error message:
Rank 1 [Mon Nov 9 14:39:00 2020] [c5-0c0s15n1] Fatal error in PMPI_Type_size: Invalid datatype, error stack:
PMPI_Type_size(119): MPI_Type_size(MPI_DATATYPE_NULL) failed

This is valid MPI; the sendtype argument (MPI_DATATYPE_NULL in the preceding example ) should be ignored when MPI_IN_PLACE is used.
I suppose the "right" solution is for IPM to ignore the sendtype, too.
A second "fix" that might be useful in other senarios is to make IPM recognize MPI_DATATYPE_NULL.

Hi,

I'm trying to build IPM on Cori but I get the following error multiple times when running 'make': error: user-defined literal operator not found.

I ran this command: ./configure --prefix=$SCRATCH/hpca/IPM MPICC=CC.

And these are the module I have loaded:

Currently Loaded Modulefiles:
  1) modules/3.2.11.4                                 14) xpmem/2.2.20-7.0.1.1_4.8__g0475745.ari
  2) nsg/1.2.0                                        15) job/2.2.4-7.0.1.1_3.34__g36b56f4.ari
  3) altd/2.0                                         16) dvs/2.12_2.2.156-7.0.1.1_8.6__g5aab709e
  4) darshan/3.1.7                                    17) alps/6.6.57-7.0.1.1_5.10__g1b735148.ari
  5) intel/19.0.3.199                                 18) rca/2.2.20-7.0.1.1_4.42__g8e3fb5b.ari
  6) craype-network-aries                             19) atp/2.1.3
  7) craype/2.6.2                                     20) PrgEnv-intel/6.0.5
  8) cray-libsci/19.06.1                              21) craype-haswell
  9) udreg/2.3.2-7.0.1.1_3.29__g8175d3d.ari           22) cray-mpich/7.7.10
 10) ugni/6.0.14.0-7.0.1.1_7.32__ge78e5b0.ari         23) craype-hugepages2M
 11) pmi/5.0.14                                       24) cray-fftw/3.3.8.5
 12) dmapp/7.1.1-7.0.1.1_4.43__g38cf134.ari           25) gcc/8.3.0
 13) gni-headers/5.0.12.0-7.0.1.1_6.27__g3b1768f.ari

Thanks,
Giulia