taskManeage - Tasks: task #15772, R (r-cran) packages in Maneage

 
 

You are not allowed to post comments on this tracker with your current authentication level.

task #15772: R (r-cran) packages in Maneage

Submitter:  Boud Roukema <boud>
Submitted:  Fri 11 Sep 2020 12:56:07 AM UTC
   
 
Should Start On:  Thu 10 Sep 2020 12:00:00 AM UTC Should be Finished on:  Thu 10 Sep 2020 12:00:00 AM UTC
Category:  Software Priority:  5 - Normal
Status:  Done Privacy:  Public
Assigned to:  boud Percent Complete:  100%
Open/Closed:  Closed Effort:  0.00

Jump to the original submission

Tue 04 Jan 2022 04:43:35 PM UTC, comment #20: 

Thanks for the confirmation, then I am closing this task. We can discuss adding more R packages during the regular software updates (with other software packages) and of course, fix any bugs that we find as bugs.

Thanks also for the interesting quote: that was an impressive prediction! I should also have a look at it ;-)!

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 04 Jan 2022 04:39:56 PM UTC, comment #19: 

Your fix looks correct to me. I like Section 3.1 of the paper:

> In addition, we decided to develop for an environment that provided two
> or more megabytes of directly addressable memory. Machines of this type are now the
> rule rather than the exception in computing.


Moore's law for a doubling time of 1.9 years over 26 years
would give about 2 Mb * (2^(26/1.9)) = 30 Gb RAM in 2021.
Impressive!

I see a hypothetical bug in the configure script, but that's a separate issue.

This task looks done to me and OK to be closed.

Boud Roukema <boud>
Group Member
Tue 04 Jan 2022 01:34:23 AM UTC, comment #18: 

Thanks Boud, the citation entry has been merged. But after trying it, I noticed that the DOI link (https://doi.org/10.2307/1390807) linked to JSTOR, not the main journal. So after searching, I found the journal DOI: https://doi.org/10.1080/10618600.1996.10474713 . The updated citation is done in Commit 7bb37e586.

About checking the citations of other packages, I just defined bug #61761.

Is there any other thing left un-done in this task?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 03 Jan 2022 08:41:53 PM UTC, comment #17: 

comment #16:

> While doing some other tests, I also noticed that without python, ICU will crash, so in the commit where I slightly edited your commit, I also added Python as a prerequisite of ICU in 6fafc7453.


Good work - I hadn't noticed that. So R depends on icu which depends on the python core!

I added the reference for the base R system:

https://gitlab.com/maneage/project-dev/-/commit/769ed43480b24e8d0920f9321015ff6b12ffd25a

I'll leave the individual packages as a task to do later by any volunteer (me or other).

Boud Roukema <boud>
Group Member
Mon 03 Jan 2022 07:30:47 PM UTC, comment #16: 

Thanks a lot for the nice hack Boud! I am happy this problem was found early and fixed :-). I have merged it into 'version-update'.

While doing some other tests, I also noticed that without python, ICU will crash, so in the commit where I slightly edited your commit, I also added Python as a prerequisite of ICU in 6fafc7453.

About the citation, we promise to add the citation of all software that have one ;-)!

Adding a citation to a package is very easy, just put the BibTeX text format in the 'reproduce/software/bibtex/' directory (following one of the existing ones; with a copyright for example), then in the build recipe, just add a single line to copy this file in the proper place in the build directory. You can look at the recipe for Gnuastro for example ;-).

Until we fix the full automation of software building, this is one extra step that we have to do for each package we add. So it would be great if you can have a fast look!

If its not written on explicitly (under a title/menu like "citation" or "publications", or similar) on the first page of the software's webpage, you don't have to search any more. Users of the software will likely know and will inform us later to add as a bug (as the maintainers of we don't have infinite time to predict and fix all bugs a-priori!).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Mon 03 Jan 2022 02:46:03 PM UTC, comment #15: 

comment #13:

> By the way, can you please also make sure if any of the R packages come with a paper to cite?


Starting at the R Journal very first issue, https://journal.r-project.org/archive/2009-1 , and 'Facets of R' by 'John M. Chambers', and https://www.r-project.org/contributors.html which says that "R was initially written by Robert Gentleman and Ross Ihaka", I think that


author = {{Ihaka}, Ross and {Gentleman}, Robert},
title = "{R: A language for data analysis and graphics}",
journal = {J.Comput.Graph.Stat.},
year = {1996},
volume = {5},
pages = {299-314},
doi = {10.2307/1390807},
issn = {10618600},


would be the best justified for giving proper credit to R as a whole. For individual R packages, that's the sort of thing that will need more automation using metadata...

Boud Roukema <boud>
Group Member
Mon 03 Jan 2022 02:26:25 PM UTC, comment #14: 

comment #11:

> Today, I also tried both of them again (from scratch) and again there was no problem.


The bug is now explained. To reproduce the bug:

First revert to the older version of icu (67.1):

git checkout R_4.1.2_minor_fixes # commit 64437d7
rm -f .build/software/installed/lib/libicu*
rm .local/version-info/proglib/icu-* .local/version-info/proglib/r-cran-* .local/version-info/r-cran/*


TODO by hand: enable top-level-r-cran    = r-cran-cowplot r-cran-gridExtra in TARGETS.conf


./project config --existing-conf


and then try upgrading without removing the old libraries:

git stash
git checkout version-update # commit 64deae4
git stash pop # enable top-level-r-cran    = r-cran-cowplot r-cran-gridExtra in TARGETS.conf


This should reproduce the upgrade bug - similar to https://paste.debian.net/1225582

POSSIBLE SOLUTIONS:

There's a old discussion of a similar error in ICU bug reports [1]. However, removing an old install by hand will not prevent the error for someone who upgrades in the future.

Adding in 'make clean' in 'high-level.mk' does not solve the bug. The probable reason is that 'make clean' only removes libraries in the 'icu' build directories, but not from the Maneage 'lib' install directory.

I'm not sure what the cleanest solution would be, because we haven't seen this bug in other packages, as far as I know.

I've proposed commit 7f5ac28 that branches off version-update [2]. This does a fairly specific set of =rm= commands to minimise the chance of removing a future library such as, e.g. 'icube' that starts with the same three letters.

What is clearly not a solution would be to say that people always have to install from scratch. There is no justification in increasing the CO_2 footprint when an efficient solution is known.

[1] https://unicode-org.atlassian.net/browse/ICU-20930

[2] https://gitlab.com/maneage/project-dev/-/commit/7f5ac283826b6e254d913b83a808025ee2b70938

Boud Roukema <boud>
Group Member
Sun 02 Jan 2022 11:14:40 PM UTC, comment #13: 

By the way, can you please also make sure if any of the R packages come with a paper to cite?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 02 Jan 2022 11:12:20 PM UTC, comment #12: 

For example, where you using Maneage's GCC or your host GCC?

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 02 Jan 2022 11:06:40 PM UTC, comment #11: 

Thanks for the check Boud!

Which hardware/OS/compiler were you using?

I tried it last night (before merging), and it built successfully, both on Arch GNU/Linux and Debain (within a Docker container), on an x86_64 system.

Today, I also tried both of them again (from scratch) and again there was no problem.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sun 02 Jan 2022 11:47:14 AM UTC, comment #10: 

In principle, all those changes look OK. :)  In practice, however...

Commit 64deae4 has a fatal error in compiling at least one compile instruction of icu-70.1 - probably due to an error of placing -L and -l library options in the wrong order in the compile command.

See the log at https://paste.debian.net/1225582
(Lines 1784 and 1804 are very long lines - either scroll across or download the text.)

I've left this in task #15772 for the moment. Strictly speaking it's a bug, not just a task, but hopefully it will get solved soon...


Boud Roukema <boud>
Group Member
Sun 02 Jan 2022 04:34:07 AM UTC, comment #9: 

Thanks a lot Boud!

After checking out to the branch, I was able to build it successfully and all the packages were properly installed!

Since I had the tarballs, I generalized the simple command in task #15699 (as a script that is now in comment 15), and was able to unify their formatting within a few seconds :-).

I then pushed the tarballs into our project on archiving software tarballs (as Commit 81a8cdfe29 (I mistakenly wrote 20, while it should have been 30!) and made the following two commits after your commit in that branch (since they will ultimately be squashed into all other version-update commits, I am putting their text in the P.S. Please have a look at them ;-).

With these changes, I was able to build and install the updated ICU, R and all the 30 R packages within Maneage. I have therefore already merged it with the 'version-update' branch of task #16103 and am therefore closing this task :-).

If there is any remaining issues on the installation, we can discuss it in task #16103.

It was a great effort Boud, thanks a lot!

On the 'one-letter evil' topic, I personally feel having too long variable names dramatically decreases the readability of the code). So I would have preferred a simple 'r-' prefix (after all, these variables will almost exclusively be in 'r-cran.mk'; where they are repeated a lot!). But you are the maintainer of the R packages, so I respect your choice and won't complain ;-).

P.S.


commit 2ccbc6f150b7f4ae81801ae3aae65bcc324cf644
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Date:   Sun Jan 2 01:56:20 2022 +0100

    Minor formatting edits in r-cran.mk

    In order to fit the formatting and style of the rest of Maneage's source,
    the formattings below are implemented in this commit:

     - When using pipes or '&&' and the line needs breaking (with a '\'), the
       '\' is placed before the '|' or '&&', so the next line starts with these
       tokens. This greatly helps in readability (because they can be missed at
       the end of a long line).

     - Emacs's 'M-q' (ALT + q key) was pressed on the multi-line comment
       paragraphs to justify the lines with a standard width to all comments.

     - The 'version-info/r-cran' directory is now made in
       'reproduce/software/shell/configure.sh', like all other similar files
       and removed from the recipe for 'R'.

     - Generally, where possible we should break lines that are longer than 75
       characters, so where necessary, the prerequisites been moved to a new
       line.

     - For the text that goes into the LaTeX macro to be printed in the final
       PDF (not as an internal variable), I have removed the common 'r-cran',
       because it will not generally be recognized and may cause
       confusion. Ideally, it should be the official name of the package, on
       its webpage in basic English, for example also respecting the large-caps
       letters in its name.

     - When the number of independent lines that are immediately following each
       other (for example the list of prerequisites when there are many of
       them), it is much more human-friendly for them to be sorted by
       length. The order of the prerequisites doesn't affect our recipes, so
       its best to do this and help the human eye when inspecting the code ;-).

commit 64deae4ce04b4c1a1f9567348bcc9b573e8f5649
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Date:   Sun Jan 2 04:53:06 2022 +0100

    Included R package tarballs are now in Maneage format

    The included R packages have been formatted in the Maneage '.tar.lz'
    format, and those tarballs are now being used. In the process, the
    following generalizations also have been done to the 'r_cran_build'
    function:

     - The indentation has been fixed.

     - Generally, all the variables defined in this function open inside the
       recipe of rules here, so there is no chance of conflicting with other
       variables. As a result (to have a significant help in readability!), the
       'r_cran_' prefix has been removed from the variable names.

     - By definition, none of these variables will also have a space or white
       character! So again, to help in readability, the '$${}' was just changed
       to '$$'. Where necessary (one instance, since '_' is used) double-quotes
       have been put around them.

     - As explained in the comments, since R's 'install.packages' only accepts
       '.tar.gz' files, we now unpack the tarball before calling R, and give it
       the directory (this works for tarballs from R's server and Maneage's
       tarballs).

     - The extra 'versioncheck' has been removed.

    To follow a similar convention to other software's build rules, the
    'tarball' variable is now defined at the start of the recipe of each R
    package to let the users customize it as necessary and per-package
    (especially when not using Maneage's archived tarball). A comment has been
    added, explaining what to do.

    Furthermore, the version of ICU has also been updated and its tarball is
    now also in Maneage.


Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 01 Jan 2022 11:54:45 PM UTC, comment #8: 

Commit 64437d76 (branched directly off 0b6e88dc in the "R" branch) has two minor fixes and removes the old rule for R. This fully works for me (including producing the pdf), and with

top-level-r-cran    = r-cran-cowplot r-cran-gridExtra

enabled 'configures' the libraries fully.

https://gitlab.com/maneage/project-dev/-/commit/64437d767a210bb536140320a62ca5f6796e834b

Regarding the one-letter evil :P, I think that writing "R" in sentences is generally OK, but keywords such as variable names and message titles should preferably have R-CRAN or r-cran, following Debian, which has r-base for the base package and r-cran for the libraries. I don't object to following Debian more precisely, i.e. separating the base package from the libraries, but 'cran' is a much more unusual string than 'base', so including the base together with the libraries, all as 'r-cran', probably doesn't hurt for the moment. If someone wants a rename to separate 'r-base' from 'r-cran', then I don't see a strong objection. But I would object to reverting to the single-letter name: grepping and other searches should not be unnecessarily made complicated.

Boud Roukema <boud>
Group Member
Sat 01 Jan 2022 04:07:10 PM UTC, comment #7: 

Thanks Boud, I just committed it over the rebased branch (as Commit 0b6e88dc).

It would be great if you do further updates on the inclusion of R packages over this branch (there were several minor conflict during the rebase).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Sat 01 Jan 2022 12:06:15 PM UTC, comment #6: 

You're right that 'r-cran.mk' was forgotten! It's here:

https://codeberg.org/boud/gevcurvtest/src/branch/gevcurvtest/reproduce/software/make/r-cran.mk

> to this fork of the R branch?


To which fork of the R branch would you prefer that I add the file - https://codeberg.org/boud/maneage_dev/src/branch/R_cran_icu ? version-update still seems to be at commit 5255a48.

Boud Roukema <boud>
Group Member
Sat 01 Jan 2022 04:16:09 AM UTC, comment #5: 

Thanks for all the great work on this Boud!

I rebased the two commits over the 'version-update' branch, but couldn't find any rules for all the R packages added! Reading through the message of Commit 3ee3e321f6 (hash after rebasing), I noticed that there is suppose to be a 'r-cran.mk' file that was apparently forgotten in the commits.

Can you please add the file to this fork of the R branch?

Also, could you please format the tarballs in the Maneage convention for longevity (as described in task #15699)? You can use the latest box of code in that task as the recipe (it is designed to be easily placed into a 'for' loop and done over many tarballs in cases like this).

Once the new tarballs are made, you can update the checksums and push them to a development branch of tarballs-software project, so I later merge them in (I have already added you as a "developer", so you can push branches there and avoid making a large fork!). Keeping a clone of this repository on your computer is also good to provide as the "software" directory in all your Maneage'd projects and avoiding to have many copies of the tarballs (symbolic links are placed in the build directory of each project). Alternatively, you can put the tarballs in a certain URL that I can pick-up and put in that repository myself.

As described in task #15699, it is very important for longevity that we have a unified format for all the newly added software tarballs ;-)! We can't do anything for the old ones, but all new ones should have this format.

In the process of re-building the tarballs, I also recommend to standardize the unpacked directory name to avoid complicating the un-packing+building code (and further unifying the format of the tarball).

Mohammad Akhlaghi <makhlaghi>
Group administrator
Tue 12 Oct 2021 09:22:05 PM UTC, comment #4: 

I've added the text 'r-cran' in the title of this task to make it easier to find using searches on strings.

There are now 2 commits 1ea9 and f6e2 (feel free to squash them together) for the R-cran branch:

https://codeberg.org/boud/maneage_dev/compare/maneage...R_cran_icu

The second one parallelises the icu compile, which otherwise is very slow.

1ea9: https://codeberg.org/boud/maneage_dev/commit/1ea9d6db9fab5fd2d88fb51d149777a8090b038f

f6e2: https://codeberg.org/boud/maneage_dev/commit/f6e284f149241194b1962e52850ce5343ca36c3f

Boud Roukema <boud>
Group Member
Fri 01 Oct 2021 02:25:24 PM UTC, comment #3: 

Please ignore https://gitlab.com/maneage/project-dev/-/merge_requests/5 and instead use this updated commit:

https://codeberg.org/boud/maneage_dev/commit/1ea9d6db9fab5fd2d88fb51d149777a8090b038f

that branches directly off commit 3a1b96766c. Commit 1ea9d6d includes a dependence on the 'icu' libraries, to avoid becoming system-dependent or failing when the icu libaries are absent.

As before, offline testing and Darwin/xnu testing would be good.

Boud Roukema <boud>
Group Member
Mon 21 Jun 2021 11:47:10 PM UTC, comment #2: 

I more or less followed the 11 Sep 2020 guideline.

https://gitlab.com/maneage/project-dev/-/merge_requests/5

Hopefully this will work fine :). Here are some TODOs in case someone wants to reduce the number of future bug reports (or annoyed users who fail to submit bug reports).

TODO:

  • Testing is needed to see if this really works offline. I didn't check this.
  • Tests on Darwin/xnu (macos) systems are needed.
  • The dependencies might not yet be quite right, since some packages have common dependencies which may have been accidentally satisfied without fully testing the tree. Installs done with different numbers of cpus may detect these errors. It seems that R dependencies/prerequisites are called 'imports', e.g. see https://cran.r-project.org/web/packages/lifecycle/index.html or https://cran.r-project.org/web/packages/pillar/index.html, so in principle this should be straightforward to check for the "current" version of any R library package.


Boud Roukema <boud>
Group Member
Fri 11 Sep 2020 06:30:15 PM UTC, comment #1: 

When you added the 'R CMD INSTALL ...' command, did it download all the dependencies as binary from the internet? The "make line" is a little ambiguous, did you add a full new Make rule (with a separate target for this package), or just added in the same rule that R is built.

I have never needed R, but I know a lot of people/projects use it. So have always been looking for someone who needs it (and is thus motivated!), to help in adding R packages in Maneage similar to Python. Many projects use R and if we have the infra-structure to add R packages easily (like in Python or X.org), it would be great.

Following the Python/X.org model, we can define a 'reproduce/software/make/r.mk' Makefile file which will be only in charge of building R and its packages. I also have a similar model in mind for TeXLive packages in the development branch (with a 'reproduce/software/make/tex.mk'). We can then "include" these language-specific packages into 'high-level.mk' the same way that we add Python and X.org's packages.

Having a look at the ggplot2 page you sent, it seems the dependencies are under the "Imports" title (in this case "digest, glue, grDevices, grid, gtable, isoband, MASS, mgcv, rlang, scales, stats, tibble, withr"). The page also has the link to the "package source" tarball. So effectively it has all the components we need!

If you can find the relevant command to disable internet connection when building from the tarball, here is an "easy" way to climb down the dependency tree! This is what we did in the first days of adding Python packages:

You simply add the rule to install your high-level package, 'ggplot2' in this case, but without any internet. Once it doesn't find its dependencies, it's build should fail, saying that you need to have XXXX (if it lists many depenendencies, just take the first one).

You then add another Make rule for XXXX, and set it as a prerequisite of 'ggplot2'. In the build of XXXX, it will probably complain about not having YYYY (as a dependency of XXXX), so you add a rule for YYYY and set it as a dependency for XXXX. You follow this loop until the first package (lowest-level package!) successfully builds!

Make will then automatically come up the dependency tree you have already setup, and will complain on the first package that needs another dependency (a different branch in the dependency tree!). So again you continue and soon you'll have all the low-level infrastructure built :-).

Like Python (and any high-level language), the core dependencies should be the same for most packages, so once we build them for one package, it is much more easier to do it for other packages.

Mohammad Akhlaghi <makhlaghi>
Group administrator
Fri 11 Sep 2020 12:56:07 AM UTC, original submission:  

Does anyone currently have a maneage subsystem for installing R (R-project) extension packages?

Installing a "high" level extension package such as ggplot2

https://cran.r-project.org/web/packages/ggplot2/index.html

seems like it may need several dozen or who knows, maybe a hundred or so other R packages that it depends on.

The R install using the shell command line

R CMD INSTALL --library=$(ildir)/R/library $(word 1,$(filter $(tdir)/%,$^))

in a 'make' line in high-level.mk works fine for me for individual packages, but we really should have a system equivalent to that of the python system.


Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by makhlaghi (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-04 makhlaghi Open/ClosedOpen Closed
    2022-01-04 makhlaghi StatusIn Progress Done
        Percent Complete80% 100%
    2022-01-04 boud Percent Complete90% 80%
    2022-01-04 makhlaghi Percent Complete80% 90%
    2022-01-02 boud StatusDone In Progress
        Percent Complete100% 80%
    2022-01-02 makhlaghi StatusIn Progress Done
        Percent Complete80% 100%
    2022-01-01 makhlaghi CategoryNone Software
        StatusNone In Progress
        Percent Complete0% 80%
        Assigned toNone boud
    2021-10-12 boud SummaryR packages in Maneage R (r-cran) packages in Maneage
    2021-06-22 boud Should be Finished on2021-12-31 2020-09-10
    2021-06-22 boud Should be Finished on2020-09-10 2021-12-31
    2020-09-11 makhlaghi SummaryR-project / R-base R packages in Maneage

    Back to the top

    Powered by Savane 3.13-758e.
    Corresponding source code