The workaround of using serial mode for one program, whose failure in maneage on the CentOS system appeared to be related to openmpi, worked correctly.

> About the libraries from the host OS, they are installed as part of the GNU C Library. Hopefully when task #15390 is complete, they won't cause a problem any more :-).

That's what I suspected. That sounds like the best long-term solution for this problem.

> Was the package with semi-consistant failures only complaining about these?

Those were the only specific libraries I saw errors about, but most failure incidents gave no reports on which libraries failed, and only gave a segmentation fault of unknown origin.

I'm happy to check this later when #15390 is complete.

It seems that more work is still needed to compile openmpi to use libraries fully within the maneage subsystem.

I have one package that works fine with the openmpi options that are currently used in Maneage, but another package fails semi-consistently on the same host, within the same overall Maneage system.

Some of the packages that seem to have been installed from the host system (CentOS 2.6.32-754.18.2.el6.x86_64), as indicated by the error tracing messages, appear to be:

    /lib64/libc.so.6
    /lib64/libpthread.so.0
    /usr/lib64/ld-2.17.so
    /usr/lib64/libdl-2.17.so
    /usr/lib64/libm-2.17.so
    /usr/lib64/libutil-2.17.so

Openmpi is a huge package. I do not intend to try to solve this any time soon. (A workaround is to run this particular program in serial mode, which is an acceptable compromise.)

One possibility, that I might try if there's enough time, would be to update to a more recent upstream Maneage, which might solve this.

However, I think that a proper Maneage install of openmpi, and thorough testing on task schedulers like slurm, should be considered a major task.

Commit bbaa6af configures fine for me - see the commit description:

https://codeberg.org/boud/maneage_dev/commit/bbaa6af92186d7b5332062577ae8e8ef33bbc1df

The equivalent changes on a maneage branch that actually runs code with openmpi also worked fine. :)

This uses the internal compile strategy - and is a moderately big (slow) compile, like ghostscript and gnuastro. The compilation of openmpi does not, it seem, to be itself... parallelised.

A different but related hack to the main Maneage branch too - this was necessary for testing a git bundle with only the openmpi_slurm branch :

https://codeberg.org/boud/maneage_dev/commit/4a2b21fa65cc70cca9ab09f5803cd372fb9f96e8

Without this hack, the configure+make cycle had a fatal error at the line

v=$$(git describe --always --long maneage)

in initialize.mk , because the 'maneage' branch was missing.

Testing also requires adding openmpi to top-level-programs  = in TARGETS.conf .

Parallel processing using possibly non-shared memory, using MPI (message passing interface, a standard, not any particular software), is presently allowed for in Maneage using openmpi. How should we compile openmpi for reproducibility?

In practice, openmpi is normally to be used on a cluster or supercomputer on which jobs are submitted to and queued by and run (or rejected) by a free-software (hopefully) job/user manager such as Slurm: https://slurm.schedmd.com/ .

The computer on which a job is run is (in general) not the one from which a batch job is submitted to slurm, e.g. with 'srun'.

So roughly speaking, as I understand it:

• user uses srun or sbatch to submit a script X.sh to the slurm daemon on the frontend H;
• slurm queues the request, and after some time may choose one or more computers K and try to run X.sh under the user's identity on those computers K;
• the computers K each run X.sh, which can include a maneage package that compiles and runs program P, which uses openmpi to ask the host computer and Slurm which cpus/cores/threads it is allowed to use;
• the interaction between X.sh on K -> openmpi on K (precompiled library) -> host K + Slurm on H (and in some sense on K) is done through PMIx (pmi or pmi2); libevent; and hwloc .
• MPI means that data (arrays of bytes :)) can be sent/received among the computers K.

So the question is: for reproducibility, how much of the chain: openmpi -> (pmi + libevent + hwloc) do we want compiled internally within Maneage, and how much should it be based on autotools type automatic searching on the machine for the preferred default libraries?

There is no point trying to include slurm in Maneage, because the whole point is that the sysadmins managing a cluster use slurm to automatically manage a whole bunch of users - it's system-level software that the user's script has to interact with.

official guide: https://slurm.schedmd.com/mpi_guide.html#open_mpi

The official guide doesn't give much in terms of practical, up-to-date experience. Some URLs that seem useful:
Some URLs that seem useful:

https://bugs.schedmd.com/show_bug.cgi?id=5323

https://github.com/open-mpi/ompi/issues/5871

I'm trying some experiments, but any prior experience with this would help speed things up. :)