taskManeage - Tasks: task #16244, Software update 2022-2

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #16244: Software update 2022-2

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Sat 06 Aug 2022 12:20:26 AM UTC
   
 
Should Start On:  Fri 05 Aug 2022 11:00:00 PM UTC Should be Finished on:  Fri 05 Aug 2022 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  makhlaghi Percent Complete:  100%
Open/Closed:  Closed Effort:  0.00

Jump to the original submission

Sat 13 Aug 2022 09:13:05 PM UTC, comment #13: 

This round of general software update has been completed and pushed into the main Maneage branch as Commit 79ee52ed. All updated software tarballs have also been added to our Zenodo repository.

This task has therefore been completed and I am closing this task.

However, Maneage is rolling release, so any bug that is fixed will be immediately merged into the main branch. The reason we are updating software (and trying to find/fix related bugs) once a year is primarily due to a lack of time, and no urgent need to update once a new version of every software comes.

So please don't let this software update discourage any further testing or finding/fixing portability bugs. Any fix at any time is very welcome and will be immediately merged after confirmed tests.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 11 Aug 2022 03:26:04 PM UTC, comment #12: 

GCC 12.1.0 was fully built and attached I am sending the output of the command below (file #53560), with a slight modification (I removed the '-h' from 'free' so we have better resolution):


rm -f freecheck.txt && while true; do free | awk '/^Mem:/' >> freecheck.txt; sleep 1; done


The resulting plot is shown in the P.S. Note that GCC itself was uncompressed in '/dev/shm' (the ram), and that this is the total RAM usage. The slight increase at the base of the end of the plot is because it had started building other programs.

So probably keeping the previous 7GB (or maybe 8GB) size for available RAM should be enough.

P.S.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 11 Aug 2022 01:42:07 PM UTC, comment #11: 

To check the total amount of RAM usage by GCC, I am now re-building it. But a minute before the build, I started this simple infinte loop in another terminal to keep track of the total RAM used and available:


rm -f freecheck.txt && while true; do free -h | awk '/^Mem:/' >> freecheck.txt; sleep 1; done


In short, every second it will add a line to 'freecheck.txt', with the status of RAM. After GCC finishes, we can find the maximum amount of overall usage (and can even plot it) :-).

I won't be doing any major

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 11 Aug 2022 01:11:45 PM UTC, comment #10: 

Thanks a lot Raul and Boud,

Just now I finished a build on a system that still has a Debian from 2010! And only three basic software needed to be degraded (because they need more recent C++ standards): PatchELF, Binutils and GCC! Everything else built sucessfully :-).

After Maneage was fully installed, I was able to jump 12 years ahead (in terms of the software environment) by simply running './project shell' :-).

Based on this experience, I have added a small section in README.md about building on older systems

I also found some bugs that we hadn't noticed until, and they are now in Commit e6535484  of the 'update-software' branch.

Raul, as you see in the commit above, I added those software to 'TARGETS.conf' for now, so they are tested automatically in our final tests before the merge, thanks for the good suggestion ;-). I should say that they all built smoothly on the 12 year old system also ;-).

Boud, I agree with your point about having a unified script to check the RAM! It is indeed a great idea! Can you write a first version?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 11 Aug 2022 10:02:24 AM UTC, comment #9: 

comment #6:

> I just re-built Maneage and this time, I commented the deletion of the 'gcc-12.1.0' directory (that GCC was built in). So that directory remained.
>
> Afterwards, when GCC was fully built and installed, I checked the disk usage of that directory and got 7.4G:
>
> $ du -sh gcc-12.1.0/
> 7.4G gcc-12.1.0/
>
> So I am a little confused how it has taken ~12G on your system?


Firstly, 12GiB is the total RAM, including tmpfs space on /tmp, which is also a RAM disk: this is fairly common, especially for speed; and, of course, the RAM used during compiling.

Secondly, I haven't checked if gcc or basic.mk delete any of gcc's working files during the compilation process, so du -hs might only give a lower limit to "disk" usage in the gcc-12.1.0/ src+build directory.

So we may have something like:

total_ram_max = gcc_compilation_ram_max + slash_tmp_ram_needed + dev_shm_gcc_src_files_ram + dev_shm_gcc_working_files_ram_max

though these names are just off the top of my head and probably not the best.

This gets back to bug #62879 - it would probably make more sense to do all of this using a config/*.conf file rather than force the user to grep through configure.sh and basic.mk . Having key assumptions of the system hidden in code rather than unconfigurable is not modular and makes the system more likely to break. Of course, someone reproducing a project should, in principle, not have to modify any of the configurable parameters. But that would still be better than having to modify  configure.sh or basic.mk .

We could include a test for /tmp , to see if it's also on ram (tmpfs or other known ram-disk file systems), and also update each of the other three parameters after each gcc update.

#62879 https://savannah.nongnu.org/bugs/index.php?62879

Boud Roukema <boud>
Group Member
Thu 11 Aug 2022 08:38:54 AM UTC, comment #8: 

Thanks a lot Mohammad.

In addition to the high-level programs that are ready to be tested as part of that commit (bc49bfc), I also tested: 'emacs', 'imfit', 'missfits' and 'galsim'.

The 'TARGETS.conf' file is then:


top-level-programs = astrometrynet emacs gnuastro imfit missfits scamp sextractor swarp vim

top-level-python = astropy galsim matplotlib


It would be great if you can also test them before the final merge.

Raul Infante-Sainz <infantesainz>
Group Member
Wed 10 Aug 2022 11:50:58 PM UTC, comment #7: 

Thanks to Raul's tests on macOS, the default Maneage software, and some other commonly used software in astronomical data reduction have been updated and are now working on his macOS also! I tested those commits on my Arch GNU/Linux and it built successfully on commit bc49bfc. So it is merged on the 'update-software' branch.

Until the weekend, I will also do more complete tests on Debian, CentOS7 and Ubuntu containers also, and Raul will also try a clean build on his macOS.

To help include the necessary non-default software, those extra software are now in 'TARGETS.conf' on this branch. After all the tests are done, before merging with the 'maneage' branch, I will clean 'TARGETS.conf' ;-).

We have two more days until the deadline to update the software, so please continue testing, or updating any other software you may want updated ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 10 Aug 2022 09:42:02 PM UTC, comment #6: 

I just re-built Maneage and this time, I commented the deletion of the 'gcc-12.1.0' directory (that GCC was built in). So that directory remained.

Afterwards, when GCC was fully built and installed, I checked the disk usage of that directory and got 7.4G:


$ du -sh gcc-12.1.0/
7.4G        gcc-12.1.0/


So I am a little confused how it has taken ~12G on your system?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 10 Aug 2022 01:11:14 PM UTC, comment #5: 

Thanks a lot Boud! 12GiB change in GCC's checks has been merged into the branch! The explanations were also GREAT!

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 08 Aug 2022 08:46:30 PM UTC, comment #4: 

Gcc 12.1.0 needs about 12 GiB (approx 12.9 GB) of RAM, so I've proposed commit 692c1aa on branch RAM12GiB [1] to increase the size of the RAM in the test in basic.mk and to clarify the discussion there.

Commit 692c1aa [1] works for me from scratch through to the verify.tex macros and the pdf, on debian and centOS7, provided that I manually set --mem=12000M or greater in sbatch on the centOS7 system. Bug #62879 [2] explains why being unable to check for consistency in RAM disk constraints automatically is risky, and suggests how this could be fixed.

[1] https://codeberg.org/boud/maneage_dev/commit/692c1aafceda84809cc76ae4de976c874422d535

[2] https://savannah.nongnu.org/bugs/index.php?62879

Boud Roukema <boud>
Group Member
Mon 08 Aug 2022 07:50:38 AM UTC, comment #3: 

I tried to run Maneage from scratch but it crashed in the opening of the tarball of Dash.

At this moment we have our own compilation of Dash in order to avoid a problem with macOS, so the tarball is on the webpage of Mohammad. In Maneage, the version of Dash is '0.5.11-057cd65', however, in the webpage it is '0.5.11-057cd650a4'. They don't match and consequently, when downloading the tarball, a file that is not the tarball is generated and it causes an error:


lzip: BDIR/software/tarballs/dash-0.5.11-057cd65.tar.lz: Bad magic number (file not in lzip format).


Raul Infante-Sainz <infantesainz>
Group Member
Sun 07 Aug 2022 12:50:28 AM UTC, comment #2: 

Thanks a lot for the reminder!

That commit has been rebased over the 'update-software' branch (as Commit fe4188c64) followed by Commit 6e5f33e760 with some cleanup on the implementation and some other fixes that I found after building in a minimal CentOS 7 container.

I also tested on a Debian-Slim container, and it also builds on that.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 06 Aug 2022 11:22:47 AM UTC, comment #1: 

I don't understand the point of doing the update 27ff6f7..504e79d on branch update-software while retaining an unsolved bug in the basic stage of the configure step. The xz/lzma bug https://savannah.nongnu.org/bugs/?62700 is in the core tools level of Maneage. Or is the motivation that 504e79d on branch update-software is only a side branch that is not yet merged in the core maneage branch?

As I said over at 62700, "Commit 25539cd69c of branch 'xz_centos7_fix' of 'Maneage' [2] implements the known hack and works for me on a CentOS Linux 7.9.2009 system and a Debian GNU/Linux 11.3 system."

[2] https://codeberg.org/boud/maneage_dev/src/branch/xz_centos7_fix

Boud Roukema <boud>
Group Member
Sat 06 Aug 2022 12:20:26 AM UTC, original submission:  

It is almost one year since the last general Maneage software update (that was done in the autumn of 2021; see task #16103).

To start this round of updates, I just updated almost all of the basic software with a version increase, and some of the default ones. The job is being done on the 'update-software' branch of the development repository:

https://gitlab.com/maneage/project-dev/-/tree/update-software

Currently there is only one commit (Commit 504e79d3cb), the commit message details all the version updates and is in the P.S. of this post. I will hopefully merge this branch into Maneage in one week (Saturday April 13th 2022). It would be great if you can test this branch on your systems and help fix any bugs that you may find. Also, in case there are software you would like to update, please feel free to add them as commits over this branch until the deadline ;-).

As mentioned in the first comment of task #16103, updating non-default Maneage software needs to be done by those who use/need them (in particular Python or R packages, which have a completely different, and evolving build system). I will be happy to test and ultimately merge them once they build on a sufficient number of systems. But due to time constraints, I can't update them myself.

P.S. Commit message of first set of software updates in this round.


    IMPORTANT: Updates to many default software

    New versions of many of the Maneage software had come since the last major
    update in Maneage (almost one year ago)! With this commit, the software in
    the P.S. have been updated, and a new software (PLplot) has been added.

    Besides the updates in the core software, the followimg improvements have
    also been implemented in this commit:

     - When you run './project shell':

       - A welcome message is printed that will remind the caller that they
         have entered a new environment, it will print the location of 'HOME'
         and the location of the shell startup file.

       - The 'reproduce/software/shell/bashrc.sh' is loaded as a startup
         file. This allows you to customize your interactive Maneage shell. A
         default step has already been placed there that will put the git
         branch name (in green) within the shell prompt (which was
         purple). This greatly helps when dealing with directories under Git
         version control. These settings won't bother with Maneage's default
         operations: through environment variables we make sure that these
         './project shell' features will not slow-down the calls to the shell
         within the non-interactive Make calls.

       - The host's 'COLORTERM' is passed to the Maneage environment. It is
         used by some programs that can have color outputs on the terminal.

     - The host's 'ldd' is linked inside the custom environment. This belongs
       to the GNU C Library (which is not yet installed in Maneage). But helps
       in checking the linking status of the binaries.

     - The Zip and Unzip programs have been moved as high-level software that
       have to be manually requested when necessary. This is because they are
       not used by any of the basic software anymore. They were just installed
       as dependencies of GNU Tar to be close the other compression
       programs. Also, in the past we would use the original tarballs, and some
       (for example Numpy) were distributed in Zip format. However, by default,
       we now use a custom Lzip tarball and don't need Zip or Unzip. This was
       suggested by Zahra Sharbaf and Raul Infante-Sainz.

     - Some minor edits in 'reproduce/software/shell/tarball-prepare.sh'. In
       particular the 'awk' command was effectively just replacing a '_' with
       '-', so it just uses a simple SED expression instead.

    P.S. List of software:

    Basic software:
       cURL                                7.79.1    -->  7.84.0
       Dash                                0.5.11.5  -->  0.5.11-057cd65
       File                                5.41      -->  5.42
       Git                                 2.36.0    -->  2.37.1
       GNU AWK                             5.1.0     -->  5.1.1
       GNU Bash                            5.1.8     -->  5.2-rc2
       GNU Gzip                            1.11      -->  1.12
       GNU Nano                            6.0       -->  6.4
       GNU Libtool                         2.4.6     -->  2.4.7
       GNU libiconv                        0.16      -->  0.17
       GNU Readline                        8.1.1     -->  8.2-rc2
       GNU Binutils                        2.37      -->  2.39
       GNU Findutils                       4.8.0     -->  4.9.0
       GNU Integer Set Library (ISL)       0.18      -->  0.24
       GNU Compiler Collection (GCC)       11.2.1    -->  12.1.0
       PatchELF                            0.13      -->  0.15.0
       Perl                                5.34.0    -->  5.36.0

    High-level software:
       Libjpeg                             9d        -->  9e
       Libtiff                             4.3.0     -->  4.4.0
       CFITSIO                             4.0.0     -->  4.1.0
       Ghostscript                         9.55.0    -->  9.56.1
       GNU Astronomy Utilities (Gnuastro)  0.16.1    -->  0.18
       PLplot                              n/a       -->  5.15.0
       SCAMP                               2.6.7     -->  2.10.0
       SWarp                               2.38.0    -->  2.41.5
       WCSLIB                              7.7       -->  7.11


Mohammad Akhlaghi <makhlaghi>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #53560:  freecheck.txt.gz added by makhlaghi (53KiB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by infantesainz (Posted a comment)
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-08-13 makhlaghi StatusIn Progress Done
        Percent Complete80% 100%
        Open/ClosedOpen Closed
    2022-08-11 makhlaghi Attached File- Added freecheck.txt.gz, #53560
        Attached File- Added gcc-12.1.0-ram-usage.png, #53561
    2022-08-10 makhlaghi Percent Complete50% 80%

    Back to the top

    Powered by Savane 3.14-f13d.
    Corresponding source code