taskManeage - Tasks: task #15699, Unified format for software...


You are not allowed to post comments on this tracker with your current authentication level.

task #15699: Unified format for software tarballs

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Fri 19 Jun 2020 01:14:33 AM UTC
Should Start On:  Thu 18 Jun 2020 11:00:00 PM UTC Should be Finished on:  Thu 18 Jun 2020 11:00:00 PM UTC
Category:  Software Priority:  * 5 - Normal
Status:  Postponed Privacy:  Public
Percent Complete:  80% Assigned to:  makhlaghi
Open/Closed:  Open Effort:  0.00
* Mandatory Fields

Add a New Comment (Rich Markup)

Comment Type & Canned Response:
       No canned response available


( Jump to the original submission )

Tue 18 Jan 2022 03:40:50 PM UTC, comment #17: 

The standardized tarballs (as part of the software update of task #16103) have now been uploaded to our software repository in Zenodo: https://doi.org/10.5281/zenodo.5873409 (version cf00eda; this is the first seven characters of the hash of the commit in https://git.maneage.org/tarballs-software.git ).

I have updated the "percent complete" to 80% since we still haven't added this to the documentation! But generally, all new software that are merged into the 'maneage' branch should follow this format (and that is easy with the script we have developed below in comment #15; just after some additional automation as described there)

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 03 Jan 2022 06:49:36 PM UTC, comment #16: 

> As mentioned in comment 10, since this suggestion came after this round of updates had started, we'll leave 'tarlz' for the next round ;-).

FWIW, Paul Eggert is proposing to add an option to GNU tar to make it produce about the same format now produced by tarlz except for the CRC that tarlz adds to the extended records.

Antonio Diaz Diaz <antonio>
Sun 02 Jan 2022 03:56:40 AM UTC, comment #15: 

Here is a more generalized script that will look into all the files in an 'odir' origin-directory (assuming it only contains tarballs) and converts them to the proper format, while also printing the checksum that is also necessary in Maneage.

Just note that it assumes the formatting of the R packages: 'NAME_VERSION.tar.gz' for the tarball but it can easily be even more generalized.

Once it has been sufficiently generalized/tested, we should put this script in 'reproduce/software/shell/' (while accounting for the pax issue mentioned below also).



# Script to convert all files (tarballs in any format; just recognized
# by 'tar') within an 'odir' to a unified '.tar.lz' format.
# The inputs are assumed to be formatted with 'NAME_VERSION', and only
# for the name, we are currently assuming '.tar.gz' (for the 'sed'
# command). Please modify/generalize accordingly.
# It will unpack the source in a certain empty directory (tmpunpack),
# and rename the top directory to the requested format of NAME-VERSION
# also. So irrespective of the name of the top original tarball
# directory, the resulting tarball's top directory will have a name
# formatting of NAME-VERSION.
# Copyright (C) 2022 Mohammad Akhlaghi <mohammad@akhlaghi.org>
# Released under GNU GPLv3+

# Abort the script in case of an error.
set -e

# Directory containing original tarballs (relative to running
# directory)

# Go over all the files in 'odir'.
for f in $(ls $odir); do
    nv=$(echo $f | sed -e's/_/ /' -e's/.tar.gz//')
    n=$(echo $nv | awk '{print $1}');
    v=$(echo $nv | awk '{print $2}');
    mkdir tmpunpack
    cd tmpunpack
    tar -xf ../$odir/$f
    mv * $name/
    tar -c -Hustar --owner=root --group=root \
        -f $name.tar $name/
    lzip -9 $name.tar
    rm -r $name/
    mv $name.tar.lz ../
    cd ../
    echo $(sha512sum $name.tar.lz)
    rm -rf tmpunpack

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 01 Jan 2022 03:57:42 PM UTC, comment #14: 

For now, I am packaging the source codes with the code box in Comment #9 (without using 'tarlz').

As mentioned in comment 10, since this suggestion came after this round of updates had started, we'll leave 'tarlz' for the next round ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 01 Jan 2022 12:04:37 PM UTC, comment #13: 

So what script is currently recommended for https://gitlab.com/maneage/tarballs-software contributions ? The script from Mon 01 Nov 2021 12:09:58 AM UTC, comment #9 here at https://savannah.nongnu.org/task/index.php?15699? Or an updated script using 'tarlz'?

Boud Roukema <boud>
Project Member
Wed 03 Nov 2021 05:33:33 PM UTC, comment #12: 

> in comment #1, you mentioned that ustar is the best tar format for
> futureproof-ness. But as far as I understood so far, tarlz works
> with the pax format. Won't that be an issue for longevity?

Pax is an extension on top of ustar. (Ustar is a subset of pax). What tarlz does is to produce ustar archives by default (just like 'tar -Hustar'), but tarlz adds one extended pax header for each member that requires it (for example because the file name is too long for ustar). In the case of boost, tarlz adds 46 extended pax headers in a total of 76801 members. AFAIK, this is the most efficient and future proof that can be done just now.

You can use tarlz for all archives, and it will automatically add pax headers only to those archives requiring them.

For more info see the "pax Interchange Format" section at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html
"The physical layout of the archive shall be identical to the ustar format".

Antonio Diaz Diaz <antonio>
Wed 03 Nov 2021 03:38:47 PM UTC, comment #11: 

Thanks a lot for the great clarification! It was so puzzling for me how the Boost tarball become so large!

I didn't know about 'tarlz' until now and will certainly read more about it from its webpage.

For this round of updates, since most of the software have been updated already, it may be best to stay with this existing format. But for the next round of updates (maybe in 6 months or a year), we will certainly look into tarlz.

But just one small question: previously (in comment #1), you mentioned that ustar is the best tar format for futureproof-ness. But as far as I understood so far, tarlz works with the pax format. Won't that be an issue for longevity? Or did you just propose tarlz for scenarios like Boost (which won't work with ustar)?

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 02 Nov 2021 04:21:03 PM UTC, comment #10: 

> # If 'tar' fails with 'file name is too long (cannot be split)'
> # make the following change: '-Hustar' --> '-Hpax'.

The pax format has some inconvenients. It adds 1 KiB of headers per member, which in the case of boost_1_77_0.tar increases the size from 744 MB to 822 MB uncompressed. Half of those 78 extra MB are blocks of extended records not protected by any checksum. See http://www.nongnu.org/lzip/manual/tarlz_manual.html#Amendments-to-pax-format

Have you considered using tarlz to create the tarballs? It produces portable archives with the minimum amount of pax headers required:

tarlz --solid --owner=root --group=root -9cf $name.tar.lz $name/

Tarlz can be found at http://www.nongnu.org/lzip/tarlz.html

Antonio Diaz Diaz <antonio>
Mon 01 Nov 2021 12:09:58 AM UTC, comment #9: 

In the (long!) process of updating the Maneage software and unifying the tarball format, I have found one package so far that has long file names (and fails with the ustar method): Boost.

So to have a single place for future reference, here is the updated command-set (with comments!) to use:


tar -xf $odir/$name.tar.$suffix \
    && tar -c -Hustar --owner=root --group=root \
           -f $name.tar $name/ \
    && lzip -9 $name.tar \
    && rm -r $name/

# If 'tar' fails with 'file name is too long (cannot be split)',
# make the following change: '-Hustar' --> '-Hpax'.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Wed 06 Oct 2021 07:54:35 AM UTC, comment #8: 

That is a good point, thanks Antonio. So far, this tiny script is just here and not yet in Maneage, but to avoid forgetting, here is the current version:


tar -xf $odir/$name.tar.$suffix \
    && tar -c -Hustar --owner=root --group=root \
           -f $name.tar $name/ \
    && lzip -9 $name.tar \
    && rm -r $name/

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Tue 05 Oct 2021 03:12:18 PM UTC, comment #7: 

One more thing.

> tar xf $odir/$name.tar.$suffix \

Maybe it would be more future-proof (and less confusing for unexperienced users) to use short options with tar, instead of old style options:

"The old style syntax can make it difficult to match option letters with their corresponding arguments, and is often confusing. [...] This old way of writing tar options can surprise even experienced users."

So I suggest to change the tar command above to:
tar -xf $odir/$name.tar.$suffix \

Antonio Diaz Diaz <antonio>
Tue 05 Oct 2021 09:44:13 AM UTC, comment #6: 

Thanks again Antonio, I just edited those tarballs with the '-9' option to Lzip in Commit da4d019c9 (of the software tarball project). As we see there, the sizes did in fact shrink :-).

Just for future reference, we should now use this command to convert the tarballs to our unified format (I will try to add this in the comments of the code somewhere):


tar xf $odir/$name.tar.$suffix \
    && tar -c -Hustar --owner=root --group=root \
           -f $name.tar $name/ \
    && lzip -9 $name.tar \
    && rm -r $name/

This command currently assumes that the top-level directory within the original tarball also has the same NAME-VERSION convention. If a tarball doesn't have this convention, please add an extra 'mv' command before the 'tar -c' command so it does follow that convention.

We can later generalize this into a small script to automatically parse a directory for tarballs and make all these corrections automatically ;-).

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 04 Oct 2021 06:16:24 PM UTC, comment #5: 

> Good point! You mean to use the '--best' (or '-9') option when calling lzip, right?

Yes, but '--best' is kind of deprecated; it exists mostly for backward compatibility with gzip, and is not the best lzip can do.

I suggest to use 'lzip -9', which is, for example, what automake uses.

Antonio Diaz Diaz <antonio>
Mon 04 Oct 2021 05:57:36 PM UTC, comment #4: 

Good point! You mean to use the '--best' (or '-9') option when calling lzip, right?

I'll update the existing tarballs to be compressed at the '-9' level and continue making new ones with it. Just as a summary, I will use the following command (following on the example before):

$ name=xz-5.2.5
$ tar xf orig/$name.tar.gz \
  && tar -c -Hustar --owner=root --group=root \
         -f $name.tar $name/ \
  && lzip --best $name.tar \
  && rm -r $name/

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 04 Oct 2021 05:41:42 PM UTC, comment #3: 

>       && lzip $name.tar \

If the tarballs are going to be compressed once and decompressed each time they are needed, wouldn't it be better to compress them at level 9?

Antonio Diaz Diaz <antonio>
Mon 04 Oct 2021 01:04:39 AM UTC, comment #2: 

Thanks a lot Antonio for the very good analysis, and guidance on the proper GNU Tar command to use.

The process of unifying the format of the archived tarballs has started with Commit 0ff2ae5c23 (in the Software repository) and Commit 13ef389f05c (in a development branch of Maneage). See P.S. for the commit message describing the process.

But its just a start, I'll try to go step by step and update as many of the software as possible in the coming weeks (in the end all the commits for version update will be rebased into one that will be merged with the main Maneage branch).

If anyone would like to help, please let me know when you can do it so we coordinate and don't spend time on updating the same package.


Following the discussion in Task #15699 [1], all software that come into Maneage will be archived in a unified format. To do this, the downloaded tarball from each software's webpage (or the version we already had) was put in an 'orig' directory, then the following two commands were run (for example for xz version 5.2.5):

    $ name=xz-5.2.5
    $ tar xf orig/$name.tar.gz \
      && tar -c -Hustar --owner=root --group=root \
             -f $name.tar $name/ \
      && lzip $name.tar \
      && rm -r $name/

But besides the formatting, the top directory within the tarballs is now always just the name of the tarball without the 'tar.lz'. While most software already follow that convention, some (like Zip or Unzip) didn't. So they have also been corrected.

As one exception, for now, Make has not been updated because it would have the same file name as before and we don't want to break any dependencies (by modifying it, and thus its checksum). I tried bootstrapping Make's latest commit (so the version identifier can be different), but there was a bug when building the latest version on my computer and it has been reported to its developers [2].

[1] https://savannah.nongnu.org/task/?15699
[2] https://lists.gnu.org/archive/html/bug-make/2021-10/msg00002.html

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 27 Jun 2020 06:19:14 PM UTC, comment #1: 

It is unfortunate that GNU tar still defaults to "gnu" format[1] because the features provided by the "gnu" format were implemented in a way incompatible with other archive formats. In particular the "gnu" format is incompatible with the two POSIX formats (ustar and pax(posix)).

[1] <http://www.gnu.org/software/tar/manual/html_node/Formats.html>

IMO the best format for source tarballs is ustar because it is standard, it is simple, and it usually provides all the features required. Few source tarballs include files with names longer than 256 bytes or sizes larger than 8 GiB.

For the few source tarballs requiring some feature beyond what ustar can provide, the pax format may be used. I do not recommend using pax by default because it is extensible and feature-rich, and therefore less future-proof than ustar (which is a subset of pax).

The pax format is also not as densely packed as might be possible[2], which makes desirable the development of a new format with a fixed-field layout similar to ustar.

[2] <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html>

Therefore, my choice of tar options to run GNU tar as future-proof as possible is "tar -c -Hustar --owner=root --group=root".

Antonio Diaz Diaz <antonio>
Fri 19 Jun 2020 01:14:33 AM UTC, original submission:  

Currently the software source codes are mostly stored in the same tarball that they were originally distributed: most are '.tar.gz', some are '.tar.xz', and some '.tar.lz' (mostly those I have had to re-package for special reasons), and some even in '.zip'!

Since we are moving to use our own repository for software source codes by default (task #15686), I am proposing to adopt a special storage format also.

In particular, I have been really impressed by Lzip due to its elegance (arguably the smallest program in Maneage!), and thus future-proof-ness, and excellent compression ratio for source code: a '.tar.lz' is usually almost half the size of a '.tar.gz' file. Its is such an elegant format, that you can even use its manual to write the decompression algorithm! You don't even need to have the Lzip software! Because of all this, it is also currently the first program that Maneage builds.

So I am proposing to convert all source code that goes into the official Maneage software repository to '.tar.lz' (like on Zenodo, or our own maneage.org) . Ofcourse the existing source codes will remain untouched, this proposal only affects new software to be added.

Furthermore, Antonio (author of Lzip who has guided me a lot over the last four years), mentioned that "Another possible future problem is that GNU tar does not create POSIX (ustar) archives by default (I think)".

So besides adopting a fixed compression standard, we should also decide on a good way to run GNU Tar to be as future-proof as possible.

Please share your thoughts here to hopefully converge on a good solution soon.

Mohammad Akhlaghi <makhlaghi>
Project Administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by antonio (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  • -email is unavailable- added by makhlaghi

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.





    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-18 makhlaghi Percent Complete50% => 80%
    2021-10-05 makhlaghi Percent Complete0% => 50%
        Assigned toNone => makhlaghi
    2021-10-04 makhlaghi SummaryUnified format for software source code archival => Unified format for software tarballs
    2020-06-19 makhlaghi Carbon-Copy- => Added -email is unavailable-

    Back to the top

    Powered by Savane 3.9