taskReproducible paper template - Tasks: task #15674, ArXiv anc/ directory - publication...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15674: ArXiv anc/ directory - publication checklist

Submitted by:  Boud Roukema <boud>
Submitted on:  Fri 05 Jun 2020 01:34:29 PM UTC  
 
Should Start On:  Fri 05 Jun 2020 12:00:00 AM UTC Should be Finished on:  Fri 05 Jun 2020 12:00:00 AM UTC
Category:  Analysis Priority:  5 - Normal
Status:  None Privacy:  Public
Percent Complete:  0% Assigned to:  None
Open/Closed:  Open Effort:  0.00

( Jump to the original submission)

Mon 20 Jul 2020 08:57:25 PM UTC, comment #7: 

Thanks a lot Boud! Good point! It has been merged into Maneage as Commit 2ed8a2d60b. I used this opportunity to make some corrections:

  • I also included 'tex/src' in the files to check with 'git ls-file' and added comments on why 'tar' is being used (which may be confusing on a first view).
  • I noticed that '$$curdir' is not expanding to the value defined as a Make variable before this rule. This is because its not 'export'ed. This was causing problems in the next steps. So I changed it to '$(curdir)'.
  • Since 'tar' keeps the directories, there was no more need to make the 'reproduce' and 'tex' directories in the temporary directory.
  • Finally, thanks to this we don't need to manually delete 'LOCAL.conf' and 'gnuastro-local.conf' from the archive (they are never copied in the first place). I really liked this particular point! It was always so annoying for me to manually single-out some files.
Mohammad Akhlaghi <makhlaghi>
Project Administrator
Mon 20 Jul 2020 05:49:41 PM UTC, comment #6: 

I propose commit d0eaea9cb as one step towards improving initialize.mk in the direction that seems to me to make sense for ArXiv:

https://codeberg.org/boud/maneage_dev/commit/d0eaea9cb04d56d07b2808188c4c430d7486be9b

Here's the commit description:

Project-package-contents: only distribute git-managed files

Until this commit, the `$(project-package-contents)` rules in
`reproduce/analysis/make/initialize.mk` included a line to
provide all contents, recursively, of the directory `reproduce/`
in the package for further distribution.

This could potentially lead to the distribution of private
working files that are used during development and not intended
for general distribution.

With this commit, only those files in `reproduce/` that are
included in the current git commit are copied to the temporary
directory that is later used for creating an archive.

This commit branches off c18c170 .

Boud Roukema <boud>
Project Member
Sun 07 Jun 2020 03:46:58 AM UTC, comment #5: 

This is great! Then we won't need much of the fine-tuning that we currently do to build the tarball (in the of 'project-package-contents' of initialize.mk.

Then I propose we make a new distribution target, maybe called 'dist-arxiv' and do this job independently there (currently just building a 'tar.lz'), leaving all existing 'dist-*' targets intact (not using the option I proposed below). I look forward to using it in the second version of arXiv:2006.03018 ;-).

If you get the chance, it would be great if you could push the branch, if not no problem, I will definitely try it out (and hopefully use it) before the second version of arXiv:2006.03018.

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Sat 06 Jun 2020 02:11:06 PM UTC, comment #4: 

comment #3:

> I wasn't familiar with latexpand! It looks so interesting and useful for this purpose!


I thought I had already proposed a pull request for creating
paper-full.pdf , but probably I forgot to. The key lines are

          && sed -e 's/\({http[^ ]*\)\(%\)/\1\\%/g' paper.bbl > paper.bbl.bak \
          && latexpand --expand-bbl paper.bbl.bak $$p/paper.tex | \
            sed -e 's/\\bibliographystyle[^ }]*}//' > paper-full.tex

where the first sed line is to for ADS URLs for Astronomy & Astrophysics, that have A%26A as the ampersand hexadecimal encoded in them, and the `latexpand` line not only includes all
the include and input files, but also includes the .bbl file.

The third line, with sed , should not affect the LaTeX processing,
but is intended to avoid confusing the human reader (bibtex/
biblatex/biber have no effect on the latexpand-ed file).

> One question: does arXiv also allow uploading ".tar.gz" files in the 'anc' directory?


I don't know. It shouldn't be difficult to start uploading a
dummy paper to test this, without continuing to the submit
stage.

> If so, we can simply have three files in the 'anc' directory: 'README.md', 'README-hacking.md' and 'maneaged-XXXXXX.tar.gz' (or any other name! the name 'maneaged' just occurred to me! but the XXXXXX can be the GIt checksum).

...

> Also, since there are only three files in the 'anc' file, there won't be any problem in terms of ordering and the viewers will always see all the files ;-).
>
> What do you think?


I think you're right :) - this seems like an elegant solution to me.

Boud Roukema <boud>
Project Member
Fri 05 Jun 2020 09:10:00 PM UTC, comment #3: 

I wasn't familiar with latexpand! It looks so interesting and useful for this purpose! The idea of having the top directory only for arXiv is also good.

One question: does arXiv also allow uploading ".tar.gz" files in the 'anc' directory? If so, we can simply have three files in the 'anc' directory: 'README.md', 'README-hacking.md' and 'maneaged-XXXXXX.tar.gz' (or any other name! the name 'maneaged' just occurred to me! but the XXXXXX can be the GIt checksum).

I think this will be less over-whealming for a random arXiv viewer than having many (probably ~50) files in many sub-directories directly visible in the arXiv abstract page.

Also, none of the Maneage files are usable when downloaded alone, its much more easier for an interested reader to just download a tarball and have everything packaged and ready to go. In the '.tar.gz' file we will be able to keep file permissions and we can simply use Git to put a full snapshot of the current commit there.

Also, since there are only three files in the 'anc' file, there won't be any problem in terms of ordering and the viewers will always see all the files ;-).

What do you think?

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Fri 05 Jun 2020 06:40:40 PM UTC, comment #2: 

There are two different useful effects of using the anc/ directory
with arXiv:

(i) it is clear to the reader of the default abs/yymm.xxxxx
URL that extra files are available, in particular, it looks like the first five files
(alphabetically, digits  first, then upper case, then lower case)
are shown and the rest as (N-5) other files;

(ii) the files in anc/ are allowed to have the executable flag set
(presumably because there's no risk to the ArXiv system of
trying to execute the files).

In the source format of 2006.03018v1, I see:

    5  -rw-rw-r--    root/root      16452 project
...
-rw-rw-r--    root/root       2963 reproduce/software/shell/git-pre-commit
   39  -rw-rw-r--    root/root       1940 reproduce/software/shell/bashrc.sh
   40  -rw-rw-r--    root/root       2377 reproduce/software/shell/git-post-checkout
   41  -rw-rw-r--    root/root       6276 reproduce/software/shell/pre-make-build.sh
   42  -rw-rw-r--    root/root      45443 reproduce/software/shell/configure.sh
   43

Effect (ii) solves the executability privilege problem if we put the
full set of files in anc/ - The
user will be able to do ./project configure without having to do
chmod u+x project and chmod u+x reproduce/software/shell/* .

With 1902.09064, I was only thinking of (ii) - I noticed (i) at some point of the
submission process, but (i) wasn't a motivation in itself.
The main motivation for (ii) was to have everything within a
single directory, with a minimum number of steps for the user.

My tentative plan for submission for any particular research paper
is to have the full maneage distribution
inside of anc/, and only paper-full.tex (after usage of latexpand)
and the tex/ figure subdirectories img/ (and tikz/ if used) in the top
directory (top = top directory of tar.gz package). Probably
README* and COPYING should also be in the top directory,
either as real files or as sym links.

Boud Roukema <boud>
Project Member
Fri 05 Jun 2020 03:36:38 PM UTC, comment #1: 

This is a very interesting idea! I didn't know about arXiv's 'anc/' directory! I really like how your paper shows some of the files on the main page! We'll implement this soon and hopefully for the second submission of this paper, we'll also have it ;-).

I think it can be implemented with an option to the './project' script to be called with the 'dist' target, maybe like this:

./project make dist --for-arxiv

When its called we can add a 'anc/' directory and put relevant things there. But given the relatively large number of files that we have, it maybe easier if we just put two symbolic links to the project's 'README.md', 'README-hacking.md' and maybe even plot data, like this one. I feel that the other files are too modular/many to be of interest individually (as available in the right-side list on arXiv). What do you think?

Mohammad Akhlaghi <makhlaghi>
Project Administrator
Fri 05 Jun 2020 01:34:29 PM UTC, original submission:  

In the pre-publication checklist [1-3] we do not yet have
a step in shifting or sym-linking files into a subdirectory anc/ for ancillary files [4] in the ArXiv step of submission for astro/phys/math/compsci papers.

We can see an example of the anc/ usage here:
https://arxiv.org/abs/1902.09064

and the non-anc/ usage in v1 of the main Maneage paper:
https://arxiv.org/abs/2006.03018v1

I think that this can be done quite easily, but it requires two
elements:
(i) some minor additions/changes in one of the .mk files
(ii) info in the README-hacking.md#publication-checklist section.
There's no point doing the work of (i) if people forget to benefit from it (ii). :)

Are there any arguments against using the anc/ structure?

A weak argument against it would be that it would require splitting up files. The ArXiv guideline [4] says "tex files should not be included in this directory" but that's should, not must not. My initial guess is that the way to do (i) would be
to create anc/reproduce/ and anc/tex/ as symlinks in the maneage
system, and use the -h option in tar when creating a tar.gz file for
uploading to ArXiv, to dereference the sym links.

The benefit is that the right-hand column of the ArXiv web entry
makes it clear that there are files available for the paper.

[1] https://gitlab.com/maneage/project-dev/-/blob/pub-checklist/README-hacking.md#publication-checklist (may disappear when the branch is merged)
[2] https://gitlab.com/makhlaghi/maneage-paper/-/issues/11
[3] https://gitlab.com/maneage/project-dev/-/issues/1
[4] https://arxiv.org/help/ancillary_files

Boud Roukema <boud>
Project Member

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-06-26 boud CategoryNone => Analysis

    Back to the top


    Powered by Savane 3.5