taskManeage - Tasks: task #15262, Source code downloads and Job...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15262: Source code downloads and Job manager

Submitter:  Mohammadreza Khellat <mkhellat>
Submitted:  Wed 01 May 2019 05:21:10 AM UTC
   
 
Should Start On:  Wed 01 May 2019 12:00:00 AM UTC Should be Finished on:  Wed 01 May 2019 12:00:00 AM UTC
Category:  Software Priority:  7 - High
Status:  In Progress Privacy:  Public
Assigned to:  mkhellat Percent Complete:  80%
Open/Closed:  Open Effort:  0.00

Jump to the original submission

Fri 02 Aug 2019 01:17:53 AM UTC, comment #28: 

Fortunately today I had some time to inspect this work and try merging it. Having logs for the downloads and builds is GREAT! Thanks a lot for doing the job Mohammad-reza.

Some minor corrections were done on the branch (as fully described in the commit message). Then I imported the `master' branch into this branch to have all the new features since this branch diverged.

All this work is now pushed onto my own fork's `makelog' branch, but its not yet merged with the master branch as described below: https://gitlab.com/makhlaghi/reproducible-paper/tree/makelog

However, while doing some tests, I noticed a big problem: if the building of a program fails, the processing continues!

Until now, we solved this problem like this: we used `&&' between the main building commands until the final target is built. Therefore, as soon as one command failed, it wouldn't go onto the rest and it would stop. But here, because we need to pipe the outputs to `tee' the final status of the command is the returned value of the last command in the pipe (the `tee', which always succeeds).

For example if the `./configure' fails, it will go onto `make', `make install', and finally build the program's main target, while it shouldn't.

One solution to this problem is when we know the system has GNU Bash: with the `-o pipefail' setting (for example by running `set -o pipefail', we can tell Bash to return a failed result if any of the pipe'e commands fail. However, for the first set of software in the template (until Bash is built), we can't assume the existence of Bash.

I searched a lot for a solution, but I don't have much time and had to stop after a few hours. In particular the `script' command seemed useful, but it messed with the terminal and parallel building, causing a crash.

So the updated branch is now available on my fork until we can find a solution to this problem.

Having download and build logs is a wonderful feature, and we have done most of the job, so I have marked it as 80% complete. I hope we can find a solution to this problem soon and merge it into the `master' branch.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 08 Jul 2019 04:18:14 AM UTC, comment #27: 

Mohammad,yeah i remember our lengthy discussions around this.

I refer you to the quote from the same text in which the master is defining what he means by complexity and being hard to read.

This code is difficult to read,” said Master Foo. β€œIt is only thinly commented, its invariants are not specified, and I see no narrative description of its architecture or internal data structures anywhere. These problems will seriously impede your cooperation with other programmers.”

Again, in our case, it goes back to the purpose of commit messages which was the first thing we discussed in these comments. There are multiple examples of plaintext contents specifically designed for terminals which satisfy the rules of "standard output" none of which is a limit on a line length.

So i place that line rule not as a low-level requirement but as a suggestion made by a friend, due to habbit. That is the main topic of our discussions here.

And one more friendy tip on the side: we have never been among those who would refer to the majority as an illustration or indication of a point! At least, i know that in your work in astronomoy you have never tried to follow such a path. So, it was weird when i saw " as many other projects " in your comment.

Again please be careful, my objection to the convention is to part of it as being described as mandatory or low-level requirement.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 10:08:38 PM UTC, comment #26: 

Great! Thanks a lot Mohammad-reza! This commit message is perfectly consistent with the other commit messages in this project (and many other projects) and nicely describes the context (purpose) of the solutions.

I really like this nice "Unix Koan" which is not unrelated to our discussion: http://www.catb.org/~esr/writings/unix-koans/prodigy.html

I will "fetch" the branch tomorrow and do a clean build with it to see how it operates. If there is no problem on my side and we also check it on a macOS, we'll merge it into the main branch.

(by the way, don't forget to update the "Percent Complete" tab on the top of the page)

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 Jul 2019 08:40:01 PM UTC, comment #25: 

All the nice suggested formatting is done, my old friend.

But not as mandatory for collaboration ;).

You know it is my job to give you a hard time and yours too. Going to bed. Talk to you soon. Catch up some times.

P.S. posting Raul: Mohammad and I always have fiery discussions and much more serious than this.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 08:14:20 PM UTC, comment #24: 

Not relevant to my comment on 75 character!! Any terminal would wrap the line!! And there is a reason for that!! So you could have lines with 80+ character length.

Please do not mix taste and visual unity with a technical requirement for collaboration. In those lines you added to th readme.... you are using the word mandatory and all over this thread you are using words such as low-level conventions and requirements!!!

That is exactly what i was talking about.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 08:02:26 PM UTC, comment #23: 

These have been the conventions we have been using until now (they just weren't actually written down, but if you run `git log' you'll see them implemented consistently).

So if you want to suggest a modification to them, please define a new task and we can talk about them there.

To see the importance of the 75 character limit, run `git log' and see the different between your commit message and the rest. The
standard/default terminal width is 80 characters and Git indents all the lines of the message (to separate basic commit meta-data like author, date and etc). So if the lines in your commit message are longer than 75 characters, the extra characters will go onto the next line and break the visual indentation (and thus easy identification of the commit message).

If you have the luxury of expanding your terminal (for example on a GUI), then you can fix it by picking up the mouse and making your terminal emulator bigger (a HUGE distraction!!!!). But you don't always have that luxury and your terminal is fixed by hardware to 80 characters (since it has been a standard for decades).

But anyway, please define a new task for suggestions in new conventions (including any answer you might have to the points I raised above) ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 Jul 2019 07:28:40 PM UTC, comment #22: 

Now getting back to the main point:
Neither of the following could be a low-level convention in formatting a commit message>>
- the limit of 75 characters per line since shell wrap the lines.
- describing the context ... by suggesting to start with until now. Well, clearly because it is a suggestion and far from a mandatory convention that needs to be in place as a condition for collaboration.

On a separate note: points 2 and 3 on the commit title should be put together as one.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 07:18:19 PM UTC, comment #21: 

As for the suggestion of discussing my disagreement with the laid down convention, there are two points:
1. It is not a laid down convention until a collective agreement is made.
2. The point has been discussed under this thread so no need to open a new task.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 07:14:04 PM UTC, comment #20: 

Haha, yes, no doubt about low-level conventions required for a collaboration.

But be very careful, there is a very critical boundary between setting up rules and laying down low-level requirements or axioms.

So the point emphasized by me as "well, the comments are written in english" was a statement to reject your false comparison made in your first comment where you compared your convention with writing in english.

We are from different languages and conversing in writing, it is required to communicate through a common writing construct. And this is a valid low-level requirement.

The same goes for your hint on extending my opposition against "conventionalizing everything" to "all standards" where i clearly said " i strongly disagree with standardizing/conventionalizing everything ". It is different than saying i strongly disagree with ALL standards/ conventiins"!!!

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 06:45:06 PM UTC, comment #19: 

In short: In a collaboration, mere understandability is not enough, we need a certain level of unity also (as well as understandability and other things).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 Jul 2019 06:41:09 PM UTC, comment #18: 

Given your disdain for standards, I am happy that you have at agreed to use English as the "standard" language ;-)!

What if the next person who wants to contribute (and thinks like you), is more comfortable in Chinese?

This is a collaboration, not a single-person project. In a collaboration, we need to respect each other, and the conventions that are already in place. We need some basic (very low-level) conventions to collaborate constructively.

If you are uncomfortable with a convention, you can define that as a "task" and others will consider it. We may indeed change an aspect if you have compelling reasons.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 Jul 2019 06:22:38 PM UTC, comment #17: 

Well, the comments are written in english and as you said well explained which means that they are understandable.

As for the other concern of causing confusion due to different formatting, well different people with different mindsets engage in a project and formatting is part of the mindset.

And as i said as someone who wanna participate in this project, i am opposing that basic standard of commit message formatting.

Furthermore, your suggested standard is not about basic formatting!!

If your concern is about the oneline log, then a basic convention would be something like this:

  • You should remove the first statements on being useful ot it is good or suggested ...*

Only your number 2 should stand in its existing state. In my opinion, the rest should be gone.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 05:52:06 PM UTC, comment #16: 

Yes, it is very well explained. The explanation isn't the problem ;-).

The problem is just the type-setting: to help reading that explanation.

In projects were multiple people collaborate, they agree to some very basic standards. For example you could have written the commit message in Persian. But by doing that, it wouldn't be useful for others.

Similarly, following those type-setting recommendations makes your commit fit nicely into the Git history: in particular the outputs of `git log' and `git log --oneline' (which are very commonly used).

Having one author's commit messages be out of format makes things confusing, and obscures the history when someone is running those commands to see/understand the project history.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 Jul 2019 05:43:21 PM UTC, comment #15: 

Hi Mohammad,
I would not do those changes or correct the commit message. In my opinion, the commit message is explanatory enough and serves the purpose i understand should be served by a commit message.

Most importantly, i strongly disagree with standardizing/conventionalizing everything. I know you understand my point of view. ;)

Of course, this doesnt mean that i would necessarily stick to the same frame of mind later on.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 05:02:32 PM UTC, comment #14: 

Great! Thanks. I am now able to see the commit.

After seeing the commit message, I noticed that we don't yet have some guidelines on the format of commit messages. It was just added to README-hacking.md.

In order to have a uniform viewing of commit messages by all authors, can you please correct the format of the commit message? I will then pull it and try it out ;-).

This is easy to do: 1) Go to this branch. 2) Run `git commit --amend' and correct the commit message so it conforms to the standards. 2) Run this command to force-push the new commit to your remote (just change `<REMOTENAME>' to the name you set for your fork): `git push <REMOTENAME> +HEAD'. But just in case there is any mistake/error, before doing those steps, be sure you have a backup of your changes on your system somewhere ;-).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 07 Jul 2019 04:17:04 PM UTC, comment #13: 

Hi Mohammad and Raul,
Apologies for the inconvenience; I had not checked the settings of the gitlab project. I added you both as Developers for the fork. Please let me know if you encountered any other issues.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 07 Jul 2019 03:09:41 PM UTC, comment #12: 

Hi Mohammad-reza, I couldn't open that page. Has it been built as a "private" project? If so, you should add me (atleast) as a member so I can access and pull from it.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 25 Jun 2019 03:31:49 AM UTC, comment #11: 

hi guys,
This is the address of the fork:
https://gitlab.com/mkhellat/reproducible-mkhellat

Mohammadreza Khellat <mkhellat>
Group Member
Sun 23 Jun 2019 11:16:18 AM UTC, comment #10: 

Great! Thanks a lot Mohammad-reza.

Can you just give the link of your fork also? This way, we can also see the progress and comment on it

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 23 Jun 2019 11:04:31 AM UTC, comment #9: 

The relevant changes as well as a new logging and messaging is available in the branch: task-tarball_downloader-15262.

Next steps would be to include integrity check for downloads as well as building tee in a similar way like flock. New tasks would be created for both.

I also found that since the download and build would be done in parallel, the configure might fail due to the failure of the downloadwrapper. Yet the user might not be able to see that because the script would continue an in-progress build.

A workaround might be to make the configure script print out the final downloadwrapper message if the configure crashes.

Mohammadreza Khellat <mkhellat>
Group Member
Mon 27 May 2019 12:24:44 PM UTC, comment #8: 

You are right! I didn't notice that you had used `--user-agent' already, thanks for the correction.

In parallel to this, can you also try GNU Guix on your operating system and share your results under that task? I have been in touch with one of the founders of Guix and he is also a recipient of the discussions under that task. I want to see if this geographic limitation also affects GNU Guix?

As you might have noticed, in the parallel task #15286, I am proposing to migrate to using GNU Guix as the template's package manager. It has a far more superior feature-set for reproducibility than ours, and has a relatively large community which are also passionate about reproducibility and can help us focus on the higher-level usage of the software in a reproducible way, rather than low-level installation of the software.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 27 May 2019 08:15:52 AM UTC, comment #7: 

Mohammad, please check my first post, you could see the --user-agent flag, there.
The problem as I mentioned is related to geographical location.

I completely agree with the backup plan...I am working on it in the downloadwrapper; would try to do that based on some debug info.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 26 May 2019 09:50:36 AM UTC, comment #6: 

Thanks for the check. Did you also try the `--user-agent' option to Wget (from my previous message on this task)?

I agree! We need to account for a backup webserver for the software tarballs. We can use the Git server. But currently you say GitLab is also blocked in Iran!!!

We can move the tarballs repository to Github as one solution. Another better solution is Zenodo. But the problem with Zenodo is that we'll need to make a new publication on every tarball!

But generally, we need to do two things:

  • Implement a backup feature in the downloading script.
  • Add an option to the download script to check tarball integrity with an MD5 sum. We already do this for the downloaded dataset, but in `downliad.mk'.


Once these are done we can easily set/change the backup srver URL any time.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 26 May 2019 09:32:33 AM UTC, comment #5: 

Okay, checked it. It is related to geoprotection. so if we wanna download with curl or wget we need an anonymizor.

I think it is best to give the option of downloading the tartballs using git.

Mohammadreza Khellat <mkhellat>
Group Member
Sun 26 May 2019 08:19:32 AM UTC, comment #4: 

Hey guy, sorry for the late reply.

Based on what I have observed, it could be related to two things:
1- the webserver performing a redirect to one of its mirrors; this should be solved with "curl -L".

2- Geographical location.

So here is what I am gonna do first:
a. would use an anonymizor to fool the servers regarding the location.
b. gonna make configure use "curl -L" to download the tarballs.

Mohammadreza Khellat <mkhellat>
Group Member
Wed 01 May 2019 09:07:59 PM UTC, comment #3: 

Just as a confirmation:

  • Do also have this problem with cURL? just for a test, you can replace `"$$downloader"' with `"curl -L -o"' in line 203 of `reproduce/software/make/basic.mk' (or `high-level.mk' and `python.mk').


  • Atleast with Wget, it is possible to make the server think that its a browser through the `--user-agent' command. For example see this StackOverflow question/answer. Can you try this (the same `--user-agent' value given in the answer? When using Wget, just add it to the value of `downloader' similar to before.


  • (Just as a side-note) Another solution for individual projects is to also upload the tarballs that they have used along with their results. For example this is what I did in one of my older projects (see `gnuastro-dependencies.tar.gz' in zenodo.1163746). I am a little reluctant to use our own Git repository as the default downloading location of the software tarballs. We don't just want to run the pipeline, it is also a kind of documentation. When the URL (a weak "Persistant Identifier") belongs to the authentic webpage of the project, a reader will know that we haven't made any modifications to it. But let's see if the points above solve this issue or not?


  • Since Mohammad-reza is the first person to report this problem and we can't reproduce it, I assigned this job to him ;-).
Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 01 May 2019 11:46:30 AM UTC, comment #2: 

You see the complication comes from the hosting web-server.
In other words, for different servers, you might need to use a different command.

The logical method is to use git repo itself; and use git commands. So i suggest we do this:

  • Instead of downloading one-by-one, lets put all the required tar balls in a git repo and then download it at the start of the configure script using git commands. However, yet I need to figure out the optimized way of updating the downloaded git repo later on.


  • After the download is done, the rest could happen as it is normally happening in the configure script.





And here is the main msg, so you would be happy ;) and it would look nicer:

The source code download issues are really annoying.

  • Only https links should be used (no ftp no http). It should be understood that web servers might have different security configurations and the best way to download files from them would be through modern web-browsers unless they have provided and documented specific wget and curl commands as part of their documentation. Here is the wget equivalent command of downloading libffi through mozilla firefox 65.0:


wget --header 'Host: gcc.gnu.org' --user-agent 'Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --referer 'https://gcc.gnu.org/pub/libffi/' --header 'DNT: 1' --header 'Proxy-Authorization: Basic Lmh4QDQ2MjI5NDtpci46Nnp4SStsYzhmYldORXZmZnZXMlg3ZVlUTGhFSkpNRFljOHFHeDdNVnY4RT0=' --header 'Upgrade-Insecure-Requests: 1' 'https://gcc.gnu.org/pub/libffi/libffi-3.2.1.tar.gz' --output-document 'libffi-3.2.1.tar.gz'








So, I could say downloading packages using wget is not the best option.

  • There should be 3 reties and at the end, the user should be informed to download the package manually, and the script should exit. The job manager should be configured in a way that configure script should continue the job from where it was hanging not from a random new place.


  • The other option is that all the required packages be hosted on a separate git repo. This would be quite standard because at least we know that the pkgs are accessible through the same method through which the main source code was accessible.
Mohammadreza Khellat <mkhellat>
Group Member
Wed 01 May 2019 10:48:11 AM UTC, comment #1: 

These are good points. Thanks ;-).

Do you propose a better program for downloading?

The Wget error is for a https link, does this mean that only relting on https isn't the solution? By the way, Savannah does have a small Markup language that is good to include outputs or commands (different from the text): see this.

The script that does the downloading is reproduce/analysis/bash/download-multi-try. You can look into it and make the proper corrections ;-).

That is a good point about a repo for software packages. Indeed, in the READMEs, we encourage each project to have such a repo. Here is the one for the raw template: https://gitlab.com/makhlaghi/reproducible-paper-dependencies

Mohammad Akhlaghi <makhlaghi>
Group administrator
Wed 01 May 2019 05:21:10 AM UTC, original submission:  

The source code download issues are really annoying.

1. Only https links should be used (no ftp no http). It should be understood that web servers might have different security configurations and the best way to download files from them would be through modern web-browsers unless they have provided and documented specific wget and curl commands as part of their documentation. Here is the wget equivalent command of downloading libffi through mozilla firefox 65.0:

wget --header 'Host: gcc.gnu.org' --user-agent 'Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: en-US,en;q=0.5' --referer 'https://gcc.gnu.org/pub/libffi/' --header 'DNT: 1' --header 'Proxy-Authorization: Basic Lmh4QDQ2MjI5NDtpci46Nnp4SStsYzhmYldORXZmZnZXMlg3ZVlUTGhFSkpNRFljOHFHeDdNVnY4RT0=' --header 'Upgrade-Insecure-Requests: 1' 'https://gcc.gnu.org/pub/libffi/libffi-3.2.1.tar.gz' --output-document 'libffi-3.2.1.tar.gz'

So, I could say downloading packages using wget is not the best option.


2. There should be 3 reties and at the end, the user should be informed to download the package manually, and the script should exit. The job manager should be configured in a way that configure script should continue the job from where it was hanging not from a random new place.

3. The other option is that all the required packages be hosted on a separate git repo. This would be quite standard because at least we know that the pkgs are accessible through the same method through which the main source code was accessible.

Mohammadreza Khellat <mkhellat>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by mkhellat (Submitted the item)
  • -email is unavailable- added by mkhellat
  • -email is unavailable- added by mkhellat
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-08-02 makhlaghi StatusPostponed In Progress
        Percent Complete0% 80%
        Assigned toNone mkhellat
    2019-08-01 mkhellat StatusIn Progress Postponed
    2019-07-08 mkhellat Percent Complete80% 0%
    2019-07-07 makhlaghi Percent Complete0% 80%
    2019-05-26 mkhellat StatusPostponed In Progress
    2019-05-01 mkhellat Carbon-Copy- Added makhlaghi
        Carbon-Copy- Added infantesainz

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code