bugManeage - Bugs: bug #62879, Maneage handling of /dev/shm does...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #62879: Maneage handling of /dev/shm does not know about slurm

Submitter:  Boud Roukema <boud>
Submitted:  Mon 08 Aug 2022 08:24:49 PM UTC
   
 
Category:  Software Severity:  3 - Normal
Item Group:  Crash Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open

Thu 11 Aug 2022 10:10:09 AM UTC, comment #1: 

See https://savannah.nongnu.org/task/index.php?16244#comment9 for related discussion: /tmp/fs, gcc compilation RAM, gcc src RAM, gcc working file RAM, all need to be taken into account.

Boud Roukema <boud>
Group Member
Mon 08 Aug 2022 08:24:49 PM UTC, original submission:  

DESCRIPTION: In both (i) reproduce/software/shell/configure.sh and (ii) reproduce/software/make/basic.mk we have tests for the amount of RAM available in /dev/shm. Firstly, these should be associated with each other in some way, since someone may not realise that there are two separate tests. Secondly, neither of these take into account RAM limits imposed by a task allocation manager such as 'slurm' on an HPC (high-performance computing) shared system of systems. These can lead to a crash because the 'slurm' RAM limit is violated and Maneage (via (i) and (ii)) cannot protect against that.

EXPERIENCE: This has lead to several crashes, with error messages while compiling gcc such as

./gt-dwarf2out.h:2673:2: fatal error: error writing to /tmp/ccch3SRR.s: Cannot allocate memory

where /tmp is also a RAM disk of type tmpfs . Although this error occurs when trying to write to /tmp, not to /dev/shm, slurm tools presumably check the total RAM used by the user's task. Increasing the --mem=... parameter given to sbatch by a few GiB prevents the crashes, and does not intervene in memory allocation to /tmp.

SUGGESTION:
(1) unify and modularise the two configure.sh and basic.mk tests, in the sense that the variables setting the required amount of memory should be set in a single location, such as reproduce/software/config/ramdisk.conf;
(2) print out this RAM disk info so that it's visible in the log file;
(3) think of some way to write an optional way for Slurm (or other task managers; Slurm is licensed as free software) to interact with Maneage.

Possibility for (3):

For example, allow a third optional environment variable MANEAGE_RAMDISK_MAX, which is used in association with the reproduce/software/config/ramdisk.conf variables to decide if it's necessary to either quit or create $(BDIR)/software/build-tmp-gcc-due-to-lack-of-space and keep going. In this case, the script submitted to a slurm tool (e.g. sbatch) should set --mem=${MANEAGE_RAMDISK_MAX} and feed MANEAGE_RAMDISK_MAX through to Maneage (and Maneage would have to allow this through into configure.sh and basic.mk .)

In this way, Maneage will not need to know what external task manager is being used; it will only know that for some reason there is an extra constraint on the ram disk than the output of the command line df /dev/shm .

This third memory constraint parameter should be documented in the .conf file.

SLURM:


Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code