I propose commit d0eaea9cb as one step towards improving initialize.mk in the direction that seems to me to make sense for ArXiv:

https://codeberg.org/boud/maneage_dev/commit/d0eaea9cb04d56d07b2808188c4c430d7486be9b

Here's the commit description:

Project-package-contents: only distribute git-managed files

Until this commit, the `$(project-package-contents)` rules in
`reproduce/analysis/make/initialize.mk` included a line to
provide all contents, recursively, of the directory `reproduce/`
in the package for further distribution.

This could potentially lead to the distribution of private
working files that are used during development and not intended
for general distribution.

With this commit, only those files in `reproduce/` that are
included in the current git commit are copied to the temporary
directory that is later used for creating an archive.

This commit branches off c18c170 .

comment #3:

> I wasn't familiar with latexpand! It looks so interesting and useful for this purpose!

I thought I had already proposed a pull request for creating
paper-full.pdf , but probably I forgot to. The key lines are

          && sed -e 's/\({http[^ ]*\)\(%\)/\1\\%/g' paper.bbl > paper.bbl.bak \
          && latexpand --expand-bbl paper.bbl.bak $$p/paper.tex | \
            sed -e 's/\\bibliographystyle[^ }]*}//' > paper-full.tex

where the first sed line is to for ADS URLs for Astronomy & Astrophysics, that have A%26A as the ampersand hexadecimal encoded in them, and the `latexpand` line not only includes all
the include and input files, but also includes the .bbl file.

The third line, with sed , should not affect the LaTeX processing,
but is intended to avoid confusing the human reader (bibtex/
biblatex/biber have no effect on the latexpand-ed file).

> One question: does arXiv also allow uploading ".tar.gz" files in the 'anc' directory?

I don't know. It shouldn't be difficult to start uploading a
dummy paper to test this, without continuing to the submit
stage.

> If so, we can simply have three files in the 'anc' directory: 'README.md', 'README-hacking.md' and 'maneaged-XXXXXX.tar.gz' (or any other name! the name 'maneaged' just occurred to me! but the XXXXXX can be the GIt checksum).

...

> Also, since there are only three files in the 'anc' file, there won't be any problem in terms of ordering and the viewers will always see all the files ;-).
>
> What do you think?

I think you're right :) - this seems like an elegant solution to me.

There are two different useful effects of using the anc/ directory
with arXiv:

(i) it is clear to the reader of the default abs/yymm.xxxxx
URL that extra files are available, in particular, it looks like the first five files
(alphabetically, digits  first, then upper case, then lower case)
are shown and the rest as (N-5) other files;

(ii) the files in anc/ are allowed to have the executable flag set
(presumably because there's no risk to the ArXiv system of
trying to execute the files).

In the source format of 2006.03018v1, I see:

    5  -rw-rw-r--    root/root      16452 project
...
-rw-rw-r--    root/root       2963 reproduce/software/shell/git-pre-commit
   39  -rw-rw-r--    root/root       1940 reproduce/software/shell/bashrc.sh
   40  -rw-rw-r--    root/root       2377 reproduce/software/shell/git-post-checkout
   41  -rw-rw-r--    root/root       6276 reproduce/software/shell/pre-make-build.sh
   42  -rw-rw-r--    root/root      45443 reproduce/software/shell/configure.sh
   43

Effect (ii) solves the executability privilege problem if we put the
full set of files in anc/ - The
user will be able to do ./project configure without having to do
chmod u+x project and chmod u+x reproduce/software/shell/* .

With 1902.09064, I was only thinking of (ii) - I noticed (i) at some point of the
submission process, but (i) wasn't a motivation in itself.
The main motivation for (ii) was to have everything within a
single directory, with a minimum number of steps for the user.

My tentative plan for submission for any particular research paper
is to have the full maneage distribution
inside of anc/, and only paper-full.tex (after usage of latexpand)
and the tex/ figure subdirectories img/ (and tikz/ if used) in the top
directory (top = top directory of tar.gz package). Probably
README* and COPYING should also be in the top directory,
either as real files or as sym links.

In the pre-publication checklist [1-3] we do not yet have
a step in shifting or sym-linking files into a subdirectory anc/ for ancillary files [4] in the ArXiv step of submission for astro/phys/math/compsci papers.

We can see an example of the anc/ usage here:
https://arxiv.org/abs/1902.09064

and the non-anc/ usage in v1 of the main Maneage paper:
https://arxiv.org/abs/2006.03018v1

I think that this can be done quite easily, but it requires two
elements:
(i) some minor additions/changes in one of the .mk files
(ii) info in the README-hacking.md#publication-checklist section.
There's no point doing the work of (i) if people forget to benefit from it (ii). :)

Are there any arguments against using the anc/ structure?

A weak argument against it would be that it would require splitting up files. The ArXiv guideline [4] says "tex files should not be included in this directory" but that's should, not must not. My initial guess is that the way to do (i) would be
to create anc/reproduce/ and anc/tex/ as symlinks in the maneage
system, and use the -h option in tar when creating a tar.gz file for
uploading to ArXiv, to dereference the sym links.

The benefit is that the right-hand column of the ArXiv web entry
makes it clear that there are files available for the paper.

[1] https://gitlab.com/maneage/project-dev/-/blob/pub-checklist/README-hacking.md#publication-checklist (may disappear when the branch is merged)
[2] https://gitlab.com/makhlaghi/maneage-paper/-/issues/11
[3] https://gitlab.com/maneage/project-dev/-/issues/1
[4] https://arxiv.org/help/ancillary_files