Exchanging data with PMIx_Publish/Lookup does not scale well about prrte HOT 6 CLOSED

Did you configure prrte with --enable-debug ? If you do not, some calls to ORTE_NAME_PRINT will be #ifdef’ed out and performance should be much higher.

from prrte.

@ggouaillardet This is my prrte configuration:

./configure --prefix=$INSTALLPATH --with-hwloc=$INSTALLPATH --with-pmix=$INSTALLPATH --with-libevent=$HOME/.local --without-slurm --with-tm=/cm/shared/apps/torque/6.1.1

I tried again and specified --disable-debug, same behavior.

from prrte.

Setting aside the debug question - the use of publish/lookup to exchange endpoint data is never going to work well as it was not intended for that purpose. If you read the standard, you will see that the exchange of data during wireup is done with the code path you eventually adopted - put/commit/fence/get. This code path is done as a collective operation and thus can scale.

Publish/lookup is intended for things like the rendezvous between two unrelated jobs where one process from each job takes responsibility for posting and retrieving the information about the other job, and then disseminating it to its peers. It is designed not to scale for the simple reason that it isn't a collective operation - thus, you can not take advantage of any of the scalable operations used in the other code path.

At some point in the near future, you will not need to put/commit any data. All endpoint information will be pre-computed and provided at startup. Thus, there will be no time required to start the job beyond just the time it takes to fork/exec the processes.

from prrte.

@rhc54 Thanks for the explanation! Closing.

from prrte.

@lyu your report contains a valid point regarding the (mostly unnecessary) cost of ORTE_NAME_PRINT()

could you please manually download and apply the patch at https://github.com/pmix/prrte/pull/226.patch ?
long story short, most ORTE_NAME_PRINT() calls will be avoided.

from prrte.

@ggouaillardet This patch works as good as manually removing the opal_output_verbose function calls!

from prrte.