taskManeage - Tasks: task #16621, Zenodo's limit on the number of...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #16621: Zenodo's limit on the number of files

Submitter:  Mohammad Akhlaghi <makhlaghi>
Submitted:  Thu 13 Feb 2025 06:50:43 PM UTC
   
 
Should Start On:  Wed 12 Feb 2025 11:00:00 PM UTC Should be Finished on:  Wed 12 Feb 2025 11:00:00 PM UTC
Category:  Discussion Priority:  5 - Normal
Status:  None Privacy:  Public
Assigned to:  None Percent Complete:  0%
Open/Closed:  Open Effort:  0.00

Discussion

Jump to the original submission

Tue 25 Mar 2025 04:09:36 PM UTC, comment #8: 

comment #4:

> Options other than Zenodo:
>
> (2) Wayback: Register an account at https:// ... hive.org and then use https://arch ... e.org/create that leads you to https://arch ... e.org/upload . Upload a bunch of tarballs and you will presumably get URLs for them all.


Done - metadata URL [1] and URL of files [2]. Only a single user has rights to delete or add files - in this case me. As long as only a tiny number of users create Wayback archives like this, it should probably be acceptable.

[1] https://archive.o ... tarballs-software
[2] https://archive.o ... tarballs-software

Boud Roukema <boud>
Group Member
Fri 21 Mar 2025 07:37:22 PM UTC, comment #7: 

About " calculating the git sha1sum with git hash-object is only one fast extra step. " As long as it is currently done (manually!), the added complexity is probably not worth it.

Generally, the philosophy of SoftwareHeritage is based on source-code, not binary blobs. So I don't feel comfortable using on SoftwareHeritage for the tarballs. It is much better to spend time on the Zenodo solution (which is designed for large binary blobs).

By the way, another complexity regarding the Git repository for software tarballs also occurred: gitlab.com has put a limit on the size of blobs. The particular blob it is complaining about is the [boost tarball](https://gitlab.ce ... ost-1.87.0.tar.lz). It was because of this that I put our institute's 'gitlab.cefca.es' as the first repository to check and can't push the 'master' branch to gitlab.com any more. At some point in the future, SoftwareHeritage may apply a similar constraint, but Zenodo is designed for such large blobs.


...
remote: GitLab: You are attempting to check in one or more blobs which exceed the 100.0MiB limit:
remote:
remote: - 7c671a80c21dd20f8760e402b31423a446d52438 (108 MiB)
remote: To resolve this error, you must either reduce the size of the above blobs, or utilize LFS.
remote: You may use "git ls-tree -r HEAD | grep $BLOB_ID" to see the file path.
remote: Please refer to https://gitlab.com/help/user/free_push_limit.md and
remote: https://gitlab.com/help/administration/settings/account_and_limit_settings.md
remote: for further information.
To gitlab.com:maneage/tarballs-software.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'gitlab.com:maneage/tarballs-software.git'


Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 21 Mar 2025 03:53:48 PM UTC, comment #6: 

It's true that the git hash-object sha1sums differ from the sha1sums of the file on its own. As SWH explains [1], the git hash is 'the SHA1 of a byte sequence obtained by juxtaposing the ASCII string "blob" (without quotes), a space, the length of the content as decimal digits, a NULL byte, and the actual content of the file':

$ wget -O astropy-7.0.1.tar.lz https://archive.softwareheritage.org/browse/content/sha1_git:3a2716ff928877e23ef8b73ca07892e303ad240e/raw/?filename=astropy-7.0.1.tar.lz && git hash-object astropy-7.0.1.tar.lz

which gives me

3a2716ff928877e23ef8b73ca07892e303ad240e


So we would have to extend from our current package-version + package-url + package-checksum scheme to also include, for those files at SWH, package-sha1git , for example.

I would recommend that these go in the same checksums.conf file. Since we already have to calculate the sha512sum of any new package, also calculating the git sha1sum with git hash-object is only one fast extra step.

An advantage would be effectively to provide an indirect extra check of the file. It would be indirect - through the URL - unless we decide to also check the sha1gitsum after downloading.

[1] https://docs.soft ... #core-identifiers

Boud Roukema <boud>
Group Member
Wed 19 Mar 2025 06:35:49 PM UTC, comment #5: 

Thanks Boud! Using SoftwareHeritage is an intriguing idea! I actually tried downloading the latest Gnuastro and Astropy's tarballs within SoftwareHeritage using wget:


$ wget -O gnuastro-0.23.tar.lz https://archive.softwareheritage.org/browse/content/sha1_git:a27ad1932fc87ed3f9a5784dec1099786066e17e/raw/?filename=gnuastro-0.23.tar.lz

$ wget -O astropy-7.0.1.tar.lz https://archive.softwareheritage.org/browse/content/sha1_git:3a2716ff928877e23ef8b73ca07892e303ad240e/raw/?filename=astropy-7.0.1.tar.lz


The commands both work! But the problem is the git hash within the URL. If it was a constant string, it would have been great, but it differs from one tarball to another (making it too complex to manage!).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 17 Mar 2025 09:23:52 PM UTC, comment #4: 

Options other than Zenodo:

(1) SWH: We do have SWH archives of the 'tarballs-software' repository:

https://archive.s ... tarballs-software (without .git)

https://archive.s ... alls-software.git (with .git)

https://archive.s ... tarballs-software (without .git)

Would using these be consistent with the 'vault' storage strategy of SWH?

(2) Wayback: Register an account at https:// ... hive.org and then use https://arch ... e.org/create that leads you to https://arch ... e.org/upload . Upload a bunch of tarballs and you will presumably get URLs for them all.

Boud Roukema <boud>
Group Member
Mon 17 Feb 2025 01:46:41 AM UTC, comment #3: 

Thanks for the comments Boud.

As in Git, Software Heritage (SWH) is designed to be optimal for source files, not binary files (tarballs). Also, generally, SWH is primarily optimal for indexing rather than retrieving: for example it is very un-optimal to use SWH as a 'git clone' alternative (since its "vault" is independent of any single repository/origin due to the many shared files in different repositories). Zenodo is designed to be optimal for the issues above: Zenodo likes large binary files rather than many small plain-text files. So the spirit of relying on Zenodo is exactly in line with leaving the technicalities to the people who are best at it ;-).

The proposals (to have separate repos based on language or alphabet) were interesting/different (I like them!). I am just worried that very soon any one of the languages or alphabetical will become more than 100 tarballs and we'll have the same problem (especially in the language classification)!

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 17 Feb 2025 12:10:31 AM UTC, comment #2: 

Ideally, Software Heritage [1] would provide frozen compressed tar sources. I vaguely seem to remember that being discussed but not being planned in the short term - their software is FOSS, so anyone willing to make a viable proposal could make a proposal there [2]. While it's OK for Maneage to do archives of source tarballs as a startup hack, it would make sense for groups doing archiving - which is a huge challenge in itself (especially in the case of backsliding democracies) - to do that properly, rather than us - in the spirit of Unix [3].

For the more immediate term: Based on a copy/paste of the Zenodo record [4] and

grep "^[a-zA-Z]" maneage.zenodo.6989624.list.20250217|grep -vE "^md5:| (kM)B"| sed -e 's/\-[0-9].*z//' -e 's/\-[0-9].*tar//' |grep -v "cert\.pem-" |sort|uniq |wc -l

it seems that we currently have slightly under 255 unique most-recent packages, ignoring version numbers. A few not-most-recent packages are still in the count, but they're just a tiny number. Removing old versions won't get us below 100 packages.

Proposal: I guess what could work is splitting up into groups of packages and have multiple Zenodo tarball collections: python and R certainly make sense being split off. Another split could be alphabetical, e.g.

  • a-e
  • f-k
  • l-p
  • q-t
  • u-z

where I'm just doing an intuitive guess on approximate frequencies of first letters in package names.

[1] https://softw ... eheritage.org

[2] https://gitlab.so ... itage.org/explore

[3] https://en.wikipe ... i/Unix_philosophy

[4] https://zenodo.or ... g/records/6989624

Boud Roukema <boud>
Group Member
Fri 14 Feb 2025 11:54:29 AM UTC, comment #1: 

They replied with this comment: You could use the API to get the desired information: https://zenodo.or ... ;allversions=true

I got the resulting page with this command:


wget -O from-zenodo.json 'https://zenodo.org/api/records?q=parent.id%3A3883409&sort=version&page=1&size=20&allversions=true'


The nice thing is that the DOI reference is in the URL, and after opening the JSON file in a text editor and searching, I did notice that even the first uploaded version (3883410) is there. We just need a tiny script to remove all the extra info and get the pure list of previous versions.

Another important point is that we need a new Zenodo repository for this new architecture so projects using the existing Maneage do not break. In the new structure, the old zenodo repository will be one of the servers to check.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Thu 13 Feb 2025 06:50:43 PM UTC, original submission:  

After the latest software update of Commit d18c167e6 (2025-02-11), I wanted to upload the standardized tarballs to [Zenodo](https://doi.org/1 ... 81/zenodo.3883409) (it is Maneage's default location to get the tarballs). However I noticed that Zenodo has recently put a limit of 100 files in a project and we already have +450 files there! I contacted their support, asking for an exception for our case.

Initially they said that you can put everything in one giant tarball! But that is not practical for our case! This is how I tried to make the case to them: The problem in this scenario is that the bulk file would be about 5GB, while any single user of our reproducible workflow system would only need 300 or 400 MB of it. For example, if you search "gnuastro" in the repo, you will see 6 different versions! Because at different points in the various projects that use this repository, different versions of Gnuastro were used. If we demand that everyone downloads everything, it will be a waste of network bandwidth (by more than an order of magnitude; which is not convenient for users who are not in Europe or have a slower/expensive internet connection compared to us in Europe) and well as storage for the users.

But in the end they said we cannot make an exception in this regard. This is a strict limit we have enforced on Zenodo for optimization. So unlike their storage limit that is possible to expand, the limit on the number of files is fixed and the latest set up updated tarballs are not on Zenodo :-(.

For the short term, we do have backups/forks on gitlab.com, git.maneage.org and gitlab.cefca.es. But these are Git repositories and not designed for high bandwidth or storage. So they should only be backups (for times like now that we don't have Zenodo). So we should find a way to address this problem!

The only thing that comes to my mind now is to not keep the old files when uploading new versions on Zenodo (that is possible!). Within Maneage, in the 'reproduce/software/shell/configure.sh' script where we check for Zenodo's base URL (from its DOI), we can also parse the side-bar and get the URLs of all the previous versions and add them to the URLs to be checked (similar to servers-backup.conf, but in the build directory, not the source). The problem with this is that the sidebar doesn't have all the previous versions, and when you click on "View all XX versions", the page isn't a simple HTML any more (to parse easily!). I just asked their support system if there is a command-line (for example Wget or cURL) interface to get the list of records associated to a high-level DOI.

In the meantime, if anyone here has any thoughts or suggestions, don't hesitate to share them here ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator

 

Attached Files

This item currently has no attached files.

 

Dependencies

This item does not depend on any other items.

No items depend on this one.

 

Mail Notification Carbon-Copy List

Carbon-Copy List
  • -email is unavailable- added by boud (Posted a comment)
  • -email is unavailable- added by makhlaghi (Submitted the item)
  •  

    Votes

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    History

    No changes have been made to this item

    Back to the top

    Powered by Savane 3.16-a7ba.
    Corresponding source code