taskReproducible paper template - Tasks: task #15699, Unified format for software source...


You are not allowed to post comments on this tracker with your current authentication level.

task #15699: Unified format for software source code archival

Submitted by:  Mohammad Akhlaghi <makhlaghi>
Submitted on:  Fri 19 Jun 2020 01:14:33 AM UTC  
Should Start On:  Thu 18 Jun 2020 11:00:00 PM UTC Should be Finished on:  Thu 18 Jun 2020 11:00:00 PM UTC
Category:  Software Priority:  5 - Normal
Status:  Postponed Privacy:  Public
Percent Complete:  0% Assigned to:  None
Open/Closed:  Open Effort:  0.00

Sat 27 Jun 2020 06:19:14 PM UTC, comment #1: 

It is unfortunate that GNU tar still defaults to "gnu" format[1] because the features provided by the "gnu" format were implemented in a way incompatible with other archive formats. In particular the "gnu" format is incompatible with the two POSIX formats (ustar and pax(posix)).

[1] <http://www.gnu.org/software/tar/manual/html_node/Formats.html>

IMO the best format for source tarballs is ustar because it is standard, it is simple, and it usually provides all the features required. Few source tarballs include files with names longer than 256 bytes or sizes larger than 8 GiB.

For the few source tarballs requiring some feature beyond what ustar can provide, the pax format may be used. I do not recommend using pax by default because it is extensible and feature-rich, and therefore less future-proof than ustar (which is a subset of pax).

The pax format is also not as densely packed as might be possible[2], which makes desirable the development of a new format with a fixed-field layout similar to ustar.

[2] <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html>

Therefore, my choice of tar options to run GNU tar as future-proof as possible is "tar -c -Hustar --owner=root --group=root".

Antonio Diaz Diaz <antonio>
Fri 19 Jun 2020 01:14:33 AM UTC, original submission:  

Currently the software source codes are mostly stored in the same tarball that they were originally distributed: most are '.tar.gz', some are '.tar.xz', and some '.tar.lz' (mostly those I have had to re-package for special reasons), and some even in '.zip'!

Since we are moving to use our own repository for software source codes by default (task #15686), I am proposing to adopt a special storage format also.

In particular, I have been really impressed by Lzip due to its elegance (arguably the smallest program in Maneage!), and thus future-proof-ness, and excellent compression ratio for source code: a '.tar.lz' is usually almost half the size of a '.tar.gz' file. Its is such an elegant format, that you can even use its manual to write the decompression algorithm! You don't even need to have the Lzip software! Because of all this, it is also currently the first program that Maneage builds.

So I am proposing to convert all source code that goes into the official Maneage software repository to '.tar.lz' (like on Zenodo, or our own maneage.org) . Ofcourse the existing source codes will remain untouched, this proposal only affects new software to be added.

Furthermore, Antonio (author of Lzip who has guided me a lot over the last four years), mentioned that "Another possible future problem is that GNU tar does not create POSIX (ustar) archives by default (I think)".

So besides adopting a fixed compression standard, we should also decide on a good way to run GNU Tar to be as future-proof as possible.

Please share your thoughts here to hopefully converge on a good solution soon.

Mohammad Akhlaghi <makhlaghi>
Project Administrator


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by antonio (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  • -email is unavailable- added by makhlaghi

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-06-19 makhlaghi Carbon-Copy- => Added antonio@gnu.org

    Back to the top

    Powered by Savane 3.6