> As mentioned in comment 10, since this suggestion came after this round of updates had started, we'll leave 'tarlz' for the next round ;-).

FWIW, Paul Eggert is proposing to add an option to GNU tar to make it produce about the same format now produced by tarlz except for the CRC that tarlz adds to the extended records.
http://lists.gnu.org/archive/html/bug-tar/2021-12/msg00021.html

For now, I am packaging the source codes with the code box in Comment #9 (without using 'tarlz').

As mentioned in comment 10, since this suggestion came after this round of updates had started, we'll leave 'tarlz' for the next round ;-).

> in comment #1, you mentioned that ustar is the best tar format for
> futureproof-ness. But as far as I understood so far, tarlz works
> with the pax format. Won't that be an issue for longevity?

Pax is an extension on top of ustar. (Ustar is a subset of pax). What tarlz does is to produce ustar archives by default (just like 'tar -Hustar'), but tarlz adds one extended pax header for each member that requires it (for example because the file name is too long for ustar). In the case of boost, tarlz adds 46 extended pax headers in a total of 76801 members. AFAIK, this is the most efficient and future proof that can be done just now.

You can use tarlz for all archives, and it will automatically add pax headers only to those archives requiring them.

For more info see the "pax Interchange Format" section at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html
"The physical layout of the archive shall be identical to the ustar format".

> # If 'tar' fails with 'file name is too long (cannot be split)'
> # make the following change: '-Hustar' --> '-Hpax'.

The pax format has some inconvenients. It adds 1 KiB of headers per member, which in the case of boost_1_77_0.tar increases the size from 744 MB to 822 MB uncompressed. Half of those 78 extra MB are blocks of extended records not protected by any checksum. See http://www.nongnu.org/lzip/manual/tarlz_manual.html#Amendments-to-pax-format

Have you considered using tarlz to create the tarballs? It produces portable archives with the minimum amount of pax headers required:

tarlz --solid --owner=root --group=root -9cf $name.tar.lz $name/

Tarlz can be found at http://www.nongnu.org/lzip/tarlz.html

That is a good point, thanks Antonio. So far, this tiny script is just here and not yet in Maneage, but to avoid forgetting, here is the current version:

odir=ORIGINAL_DIR
suffix=ORIGINAL_SUFFIX
name=PACKAGE_NAME_VERSION

tar -xf $odir/$name.tar.$suffix \
    && tar -c -Hustar --owner=root --group=root \
           -f $name.tar $name/ \
    && lzip -9 $name.tar \
    && rm -r $name/

Thanks again Antonio, I just edited those tarballs with the '-9' option to Lzip in Commit da4d019c9 (of the software tarball project). As we see there, the sizes did in fact shrink :-).

Just for future reference, we should now use this command to convert the tarballs to our unified format (I will try to add this in the comments of the code somewhere):

odir=ORIGINAL_DIR
suffix=ORIGINAL_SUFFIX
name=PACKAGE_NAME_VERSION

tar xf $odir/$name.tar.$suffix \
    && tar -c -Hustar --owner=root --group=root \
           -f $name.tar $name/ \
    && lzip -9 $name.tar \
    && rm -r $name/

This command currently assumes that the top-level directory within the original tarball also has the same NAME-VERSION convention. If a tarball doesn't have this convention, please add an extra 'mv' command before the 'tar -c' command so it does follow that convention.

We can later generalize this into a small script to automatically parse a directory for tarballs and make all these corrections automatically ;-).

Good point! You mean to use the '--best' (or '-9') option when calling lzip, right?

I'll update the existing tarballs to be compressed at the '-9' level and continue making new ones with it. Just as a summary, I will use the following command (following on the example before):

$ name=xz-5.2.5
$ tar xf orig/$name.tar.gz \
  && tar -c -Hustar --owner=root --group=root \
         -f $name.tar $name/ \
  && lzip --best $name.tar \
  && rm -r $name/

Thanks a lot Antonio for the very good analysis, and guidance on the proper GNU Tar command to use.

The process of unifying the format of the archived tarballs has started with Commit 0ff2ae5c23 (in the Software repository) and Commit 13ef389f05c (in a development branch of Maneage). See P.S. for the commit message describing the process.

But its just a start, I'll try to go step by step and update as many of the software as possible in the coming weeks (in the end all the commits for version update will be rebased into one that will be merged with the main Maneage branch).

If anyone would like to help, please let me know when you can do it so we coordinate and don't spend time on updating the same package.

P.S.

Following the discussion in Task #15699 [1], all software that come into Maneage will be archived in a unified format. To do this, the downloaded tarball from each software's webpage (or the version we already had) was put in an 'orig' directory, then the following two commands were run (for example for xz version 5.2.5):

    $ name=xz-5.2.5
    $ tar xf orig/$name.tar.gz \
      && tar -c -Hustar --owner=root --group=root \
             -f $name.tar $name/ \
      && lzip $name.tar \
      && rm -r $name/

But besides the formatting, the top directory within the tarballs is now always just the name of the tarball without the 'tar.lz'. While most software already follow that convention, some (like Zip or Unzip) didn't. So they have also been corrected.

As one exception, for now, Make has not been updated because it would have the same file name as before and we don't want to break any dependencies (by modifying it, and thus its checksum). I tried bootstrapping Make's latest commit (so the version identifier can be different), but there was a bug when building the latest version on my computer and it has been reported to its developers [2].

[1] https://savannah.nongnu.org/task/?15699
[2] https://lists.gnu.org/archive/html/bug-make/2021-10/msg00002.html

Currently the software source codes are mostly stored in the same tarball that they were originally distributed: most are '.tar.gz', some are '.tar.xz', and some '.tar.lz' (mostly those I have had to re-package for special reasons), and some even in '.zip'!

Since we are moving to use our own repository for software source codes by default (task #15686), I am proposing to adopt a special storage format also.

In particular, I have been really impressed by Lzip due to its elegance (arguably the smallest program in Maneage!), and thus future-proof-ness, and excellent compression ratio for source code: a '.tar.lz' is usually almost half the size of a '.tar.gz' file. Its is such an elegant format, that you can even use its manual to write the decompression algorithm! You don't even need to have the Lzip software! Because of all this, it is also currently the first program that Maneage builds.

So I am proposing to convert all source code that goes into the official Maneage software repository to '.tar.lz' (like on Zenodo, or our own maneage.org) . Ofcourse the existing source codes will remain untouched, this proposal only affects new software to be added.

Furthermore, Antonio (author of Lzip who has guided me a lot over the last four years), mentioned that "Another possible future problem is that GNU tar does not create POSIX (ustar) archives by default (I think)".

So besides adopting a fixed compression standard, we should also decide on a good way to run GNU Tar to be as future-proof as possible.

Please share your thoughts here to hopefully converge on a good solution soon.