Savannah Internals
This page describes inner workings of the GNU Savannah platform (web frontend + backend processes).
Some terminology
'GNU Savannah' (or just 'Savannah') is the name of the entire platform, encompassing the various hosts (see SavannahArchitecture), services (see SavannahServices), people (see SavannahTeam) and other odds and ends.
'Savane' (French for 'savannah') is the code name for the source code which runs many parts of the savannah platform (most notably: the web frontend at http://savannah.gnu.org).
While usually these are distinguished, they are sometimes used interchangeably, and should be appropriately understood by the context.
'groups' - in the savane code, projects are referred to as "groups" (and in a few cases, these are conceptually groups rather than programming projects, e.g., the Free Software Free Society Portuguese and the GNU Spanish Translation Team). The database table is named
groups
, and several php files are namedgroupXXX.php
while referring to projects.group_id=N
in urls refer to the project internal ID.Savannah Administrators are volunteers with special access permissions on the savannah frontend website (see 'becoming super-user' below), and administrator ssh access to the savannah hosts. See SavannahTeam and List of current administrators.
Project Administrators (in the context of this document) are savannah users who have administrative access to their own project. As an example, these people are project administrators for GNU AWK: https://savannah.gnu.org/project/memberlist.php?group=gawk.
Savannah source code
The source code for the php frontend and most of the perl/shell backend
scripts is available at:
http://git.savannah.gnu.org/cgit/administration/savane.git.
N.B. That's the administration
project/group, and the
git repository named savane
within it.
Warning about repositories: There are several other
repositories/clones/forks which look like the savane repositories (e.g.,
savane-cleanup,
another savane.git ).
The repository listed above is the only official one. All code running
on the savannah hosts comes from the administration
project, using
several version control systems there: git, cvs, subversion; each of
which holds different stuff.
Savannah hosts
Please see SavannahArchitecture and SavannahServices rather than repeating information here.
workflow - overview
- Users submit a new project. Savannah administrators approve the project (see ApprovingSubmission).
- Project administrators of existing project can change project settings (e.g., add a mailing list, switch from cvs to git, etc.).
- The PHP code on
frontend
updates the MySQL database, and optionally runs one-time project creation scripts. - On
vcs
anddownload
, cron jobs regularly query the database for updates, and make the necessary changes (e.g., create a new git repository for a new project). - The
cvsweb
repositories hold the html files which are the source for a project's web pages. When these are updated, a special hook calls a cgi script onwww.gnu.org
to request a website update. - A cron job on
www.gnu.org
regularly checks for update requests, andcvs updates
the requested repositories. - A cron job on
mgt0
regularly query the database for mailing list updates. It then creates the mailing lists onlists.gnu.org
. This is done becauselists.gnu.org
is managed by the FSF, not Savannah. - Publishing a new tarball release for GNU packages is performed by an automated process, explained in the GNU Maintainer's guide. It is not managed by Savannah administrators, and only mentioned here for completeness.
- GNU Bug Tracker (http://debbugs.gnu.org) is based on Debian's Bug Tracking System. It is not managed by Savannah administrators. Packages hosted on Savannah, either gnu or nongnu, can request to use the bug tracker. More information: http://debbugs.gnu.org/Using.html.
- For nongnu projects, project members with write access can upload new files to be available for download as 'releases' (e.g., http://download.savannah.gnu.org/releases/freetype/). While GNU projects can also store files this way, it is very strongly recommended that official GNU packages use ftp.gnu.org via the automated ftp uploads (see item 8, above). For details, see download section in SavannahServices. See also UserAuthentication.
- Project members with write access can push code to the repositories
on
vcs
using ssh. See 'vcs' section in SavannahServices for configuration details. See CvsGettingStarted, Svn, UsingHg, UsingBzr, UsingGit for usage information, and UserAuthentication for ssh authentication details.
User registration (frontend)
Anyone with a valid email account can register on GNU Savannah.
Public URL: https://savannah.gnu.org/account/register.php
Source code: /frontend/php/account/register.php
The
register.php
form creates a new record in the databaseuser
table, withSTATUS='P'
, and sends the user a confirmation URL (e.g.,https://savannah.nongnu.org/account/verify.php?confirm_hash=a1b2...
).The confirmation URL leads to /frontend/php/account/verify.php, which updates the user's record in
user
table toSTATUS='A'
.
Warning: the email that Savannah sends will often be considered spam by big email providers, or outright rejected.
Project submission (frontend)
Any savannah user can submit a submit a new project request. This procedure is for nongnu projects only (for gnu projects see here: http://www.gnu.org/help/evaluation.html).
- Public URL: https://savannah.gnu.org/register/
- Source code: /frontend/php/register/index.php (and see all files in /frontend/php/register/ ).
- After form submission, The php
confirmation.php
file inregister/
:- updates the database record in the
groups
table, withSTATUS='P'
. - sends a notification email to
savannah-register-public@gnu.org
. - Creates a new task in the task tracker (FIXME: how?)
- FIXME: the source code mentions something about
cookbook
andrecipes
database tables?
- updates the database record in the
- The project is in
pending
state until approved by Savannah administrators.
Project approval, creation (frontend)
Savannah admins approve and create projects after careful and manual review. Project creation is the same for GNU projects (the manual review for GNU projects is done by the GNU evaluation volunteers, not savannah admins).
- Public URL: https://savannah.nongnu.org/siteadmin/groupedit.php?group_id=N (only available to savannah admins in superuser mode).
- The URL is display as 'group administration' link when viewing a pending project ticket.
- Source code: /frontend/php/siteadmin/groupedit.php.
- In this page the following changes can be made:
- system name (i.e. 'short name') - the name will be used in URLs, as a unix group name for access control, and as directory names in various locations. Must be unique.
- status:
pending
,active
,deleted
,maintenance
,incomplete
. - license
- visibility (public/private)
These changes are written to the
groups
table. - Two special commands are available in this page as well:
- The green "V" icon: calls
groupedit.php
again with cgi parametersupdatefast=1
andstatus=A
, thus marking the project asactive
in thegroups
table, and in savannah. - The configure (screwdriver and wrench) icon: calls /frontend/php/siteadmin/triggercreation.php, which calls the necessary one-time project creation scripts.
- The green "V" icon: calls
- The
triggercreation.php
script does the following:- Sets project defaults (e.g., having a homepage, having 'downloads',
using cvs repository, using task,support,bugs trackers, using news,
etc.). This are fields such as
use_homepage=1
in thegroups
table.
The items set by default are hard-coded intriggercreation.php
. (this is one reason that by default, all new projects on savannah start with a cvs repository). These settings can later be changed by the project administrators. - Sends an email to the project administrator (based on the savannah user who submitted/registered the project).
- Sets project defaults (e.g., having a homepage, having 'downloads',
using cvs repository, using task,support,bugs trackers, using news,
etc.). This are fields such as
- FIXME:
triggercreation.php
mentions"site-specific triggers executed"
- which ones are these and where are they executed?
Project administration - general information (frontend)
Project admins can modify various items for their projects using the savannah web interface.
- Name, Short/Long descriptions
- public URL: https://savannah.nongnu.org/project/admin/editgroupinfo.php?group=N
- source code: /frontend/php/project/admin/editgroupinfo.php.
- The script modifies the
group
table.
- Project members (additional savannah users with write-access):
- public URL: https://savannah.nongnu.org/project/admin/useradmin.php?group=N
- source code: /frontend/php/project/admin/useradmin.php
- modifications are stored in the
user_group
table (a many-to-many relation table betweenuser
andgroups
tables). - uses code from /frontend/php/include/member.php.
- Project squads
- FIXME: a per-project list of multiple savannah users?
- public URL: https://savannah.nongnu.org/project/admin/squadadmin.php?group=N
- Source code: /frontend/php/project/admin/squadadmin.php.
- Permissions
- per-user permissions for website features (e.g., tracker management)
- Public URL: https://savannah.nongnu.org/project/admin/userperms.php?group=N
- Source code: /frontend/php/project/admin/userperms.php.
- Jobs
- "jobs" are postings which appear in the Contributors Wanted page.
- Public URL: https://savannah.nongnu.org/people/createjob.php?group=N
- Source code: /frontend/php/people/createjob.php.
- Select Features
- See "feature selection" section below.
- Configure Mailing lists
- See "mailing lists" section below.
- Configure Cookbook
- FIXME: document/explain cookbooks (or remove them).
- Set notifications:
- Select savannah users which will be notified upon tracker activity (e.g., new task, new support ticket, new bug report, etc.).
- Public URL: https://savannah.nongnu.org/project/admin/editgroupnotifications.php?group=N
- Source code: /frontend/php/project/admin/editgroupnotification.php.
- FIXME: document/explain trackers
Project administration - feature selection (frontend)
Project admins can enable/disable features for their projects, using the savannah web interface.
features in this context are the facilities used by the project, such as git, cvs, homepage, downloads, news, bugs/support/patches/tasks trackers, mailing lists, etc.
Each feature has a corresponding field in thegroups
table (e.g.,use_git
,use_homepage
,use_downloads
) which is set to1
or0
.Some features are associated with a url. The urls for these features are initialized to the savannah defaults at project creation, but can be later changed by the project administrators. Examples:
- If
use_homepage=1
, theurl_homepage
field defaults tohttp://www.gnu.org/software/PROJECT
(for gnu projects) orhttp://www.nongnu.org/PROJECT
(for nongnu projects). Project administrators can change this, although GNU packages should not do so. - If
use_git=1
, theurl_git
field defaults togit/?group=PROJECT
(which resolves tohttps://savannah.gnu.org/git/?group=PROJECT
, a page containing instructions about accessing git on savannah for this project). Project administrators can change this to any other URL, although they should not do so, especially not GNU packages. (The purpose of Savannah is to host source code.)
- If
Public URL: https://savannah.gnu.org/project/admin/editgroupfeatures.php?group=PROJECT (only available to project admins).
Source code: /frontend/php/project/admin/editgroupfeatures.php.
When updating features (selecting or deselecting features), the changes are saved to the database, in the
groups
table. No other changes are performed.These database changes will be picked up by the cron jobs running on other VMs (see below). This is the reasons some features take time to propagate (e.g., when a project administrator turns on
git
, the git repository is not created immediately). It would be better to do things synchronously, or at least very quickly.
Project administration - mailing lists (frontend)
Project admins can add/configure/delete mailing lists for their projects using the savannah web interface.
- Mailing lists require special handling: the mailing list server (lists.gnu.org, using GNU Mailman) is administered by the FSF, not Savannah.
- Special configuration on
lists.gnu.org
allows remote execution of shell scripts fromfrontend
to automate list creation/deletion. More info: ListServer. - Public URL: https://savannah.gnu.org/mail/admin/?group=PROJECT (only available to project admins).
- Source code: /frontend/php/mail/admin/index.php.
- The php code updates the
mail_group_list
table (no other direct changes are done by the script). New mailing lists are added as new records in the tables. Change requests to mailing lists are done by changing thestatus
field (see source code for possible values). - A cron job on
mgt0
reads the changes in themail_group_list
and executes the necessary changes onlists.gnu.org
. See "cron jobs on mgt" section below.
MySQL Database
See SavannahDatabaseSchema and SavannahDatabaseExamples.
Cron job on vcs
Some cron jobs may still be running on the old vcs VM still in use and
accessible as vcs
. The intent is to have everything moved off of
there eventually.
Stored in vcs0:/etc/cron.d/sv
:
# remake user list for outgoing mail.
*/10 * * * * root nice -n 11 sv_aliases --cron && sed -e '/# Savannah include start/,/# Savannah include end/s/\([^:]\+\): .*/\1: INVALID.NOREPLY@gnu.org/' /etc/email-addresses -e 's,This is /etc/email-addresses,DO NOT EDIT - GENERATED FROM email-addresses.,' > /etc/email-addresses_SENDER
# remake list of git repositories in each project.
*/15 * * * * root nice -n 11 /usr/src/infra/git/refresh-project-list.sh && /usr/src/infra/git/sv_cgit.pl
# remake CVSROOT/{commit,log}/info (for cvs commit email?).
31 * * * * root nice -n 11 /usr/src/infra/cvs/generate_log_accum.pl
# remake list of groups for the vcs.
35,05 * * * * root nice -n 11 sv_groups --cron --only-cvs --only-homepage --only-svn --only-git --only-hg --only-bzr
/backend/mail/sv_aliases.in (installed in
vcs0:/usr/local/bin/sv_aliases
) - This script prepares analias
-type file for for exim4 mail server, used when sending commit updates to mailing lists.- The script queries the MySQL database, extracting user names and
emails for all savannah users. Additionally, emails for 'squads'
(per-project mailing groups) are queried. The results are saved in
vcs0:/etc/email-addresses
. vcs0:/etc/email-addresses
is symlinked to the standardvcs0:/etc/exim4/email-addresses
file. Example:agn: assafgordon@gmail.com karl: karl@gnu.org
The
sed
command in the cronjob file replaces each user's email withINVALID.NOREPLY@gnu.org
, and saves the results invcs0:/etc/email-addresses_SENDER
. Example:agn: INVALID.NOREPLY@gnu.org karl: INVALID.NOREPLY@gnu.org
FIXME: where is
/etc/email-addresses_SENDER
mentioned/used? in an exim4 config file?
- The script queries the MySQL database, extracting user names and
emails for all savannah users. Additionally, emails for 'squads'
(per-project mailing groups) are queried. The results are saved in
refresh-project-list.sh
- this script updates thevcs0:/srv/git/project-list
file which is used by gitweb CGI application.- FIXME: file is
vcs0:/usr/src/infra/git/refresh-project-list.sh
, and is not in thesavane
repository. Add it? - gitweb configuration file is
vcs0:/etc/gitweb.conf
. Implemetation detail: the script does not re-generate the
project-list
from scratch. Instead, it scans/srv/git
for new git repositories, and only adds them to the file if they aren't already listed. This allows manually adding information to theproject-list
file, ensuring it will not be overwritten. The manually added information is the owner name/email for some repositories. Example:$ cat /srv/git/project-list ... administration.git Savannah+Hackers anubis.git bug-anubis@gnu.org autoconf.git autoconf@gnu.org automake.git automake@gnu.org autostrap.git Sylvain+Beucler coreutils.git Jim+Meyering ...
These names (e.g.
Jim Meyering
) will appear in the owner field when viewing the gitweb page of the project (example: http://git.savannah.gnu.org/gitweb/?p=coreutils.git).
- FIXME: file is
sv_cgit.pl
- this script updates thevcs0:/srv/git/cgitrepos
file which is used by the cgit CGI application.- FIXME: file is
vcs0:/usr/src/infra/git/sv_cgit.pl
and is not in thesavane
repository. Add it? - cgit configuration file is
vcs0:/etc/cgitrc
. Implementation detail: the script reads the
/srv/git/project-list
file (generated byrefresh-project-list.sh
, above), and for each mentioned project, writes an entry incgitrepos
. Example:$ cat /srv/git/cgitrepos ... repo.group=GNU Core Utilities repo.url=coreutils.git repo.path=/srv/git/coreutils.git repo.desc=GNU coreutils repo.readme=README.html repo.owner=Jim Meyering ...
- FIXME: file is
sv_groups
- This script creates new repositories for projects in the savannah database- reminder: when a new project is approved/created on
savannah, or when a project administrator selects a new repository
feature the
frontend
php code only updates the database. this script picks up the changes and executes them. - file is /backend/accounts/sv_groups.in.
- The script queries the
groups
table, and creates the needed repositories.
Example: For projectXXX
, ifgroups.use_git=1
, it look forvcs0:/srv/git/XXX.git
. If it doesn't exist, create it withgit init
and additional savannah-specific configurations (e.g. permissions, hooks, xattrs). - Repository initialization and configuration is delegated to the relevant perl module. See Git.pm, Cvs.pm, Bzr.pm, Svn.pm, Hg.pm, Download.pm.
- Two types of CVS repositories can be created:
cvs
(UsingCvsMakeArea
inCvs.pm
) - source code repository;webcvs
(UsingWebCvMakeAreaSavannahGNU
inCvs.pm
) - webcvs repository (content will be available on www.gnu.org/www.nongnu.org). The same routines also update existing CVS hooks. - The script also contains legacy code to create unix groups on the
local machine (using
/usr/sbin/groupadd
). This part is not used any more, since groups and users are delegated tonsswitch/libnss-mysql
(see UserAuthentication). - FIXME: What as "Area Attic" (mentioned in the code)?
- The script takes parameteres determining which items to create (e.g.
--only-git
). - On
vcs
host, the git/hg/bzr/cvs/cvsweb repositories are created. - On
download
host, the download (and legacy 'arch') are created.
- reminder: when a new project is approved/created on
savannah, or when a project administrator selects a new repository
feature the
Cron jobs on download
Some cron jobs may still be running on the old download VM still in
use but now accessible as olddownload
. The intent is to have
everything moved off of there eventually. FIXME: as of 2019-05-07,
olddownload is down.
stored in download0:/etc/cron.d/sv
:
*/30 * * * * root sv_groups --cron --only-download --only-arch
sv_groups
- This script creates new repositories for projects in the savannah database. See description forsv_groups
in the above section.
Cron jobs on frontend
frontend0:/etc/cron.d/sv_export
[non-functional] - contains the following commands:# XML data export 02 * * * * root sv_export --xml-path /var/cache/savane # Remove old data export 30 1 * * * root sv_export_cleaner --xml-path /var/cache/savane # Get list of Git repositories 3,33 * * * * root scp vcs.sv.gnu.org:/srv/git/cgitrepos /etc/savane
sv_export is an old feature which exports project data into XML.
FIXME: remove this?frontend0:/etc/cron.daily/sv_list_groups
[unfinished] - an attempt to publish an authoritative list of active projects hosted on GNU Savannah. The script contains the following code:mysql savane -B -e 'SELECT group_type.type_id, group_type.name, unix_group_name, group_name FROM groups, group_type WHERE groups.type = group_type.type_id AND groups.status="A" ORDER BY type;' \ > /var/www/overlay/cooperation/groups.tsv
FIXME: complete this nice feature, and expose it with apache configuration. Also possibly with JSON data, and more fields.
Cron jobs on mgt0
stored in mgt0:/etc/cron.d/sv
:
# Assign uidNumber and gidNumber inthe database (and rebuild cache for
# libnss-mysql-bg). 7211 was the latest group created by the old
# sv_groups cron - maybe one day we'll move everything to the 70000
# range. Then create homedirs and authorized_keys files.
*/5 * * * * root sv_assign_uid_gid --min-gidNumber=77212
# New list creation
*/5 * * * * root sv_mailman --cron
# Database cleaning
*/25 * * * * root sv_cleaner --cron
48 12 2 * * root sv_cleaner --cron --big-cleanup
# Reminder
4 4 * * * root sv_reminder --cron
25 18 * * * root sv_reminder --cron
sv_assign_uid_gid
- creates unique user ids and group ids for users/projects in the savannah system.- reminder: across all savannah hosts, unix user/group ids are managed with nsswitch/libnss-mysql (see UserAuthentication for details).
- Source code: /backend/accounts/sv_assign_uid_gid.in.
- When a new user (or project) is created on savannah, the php code
on
frontend
does not assign uid/gid. - This script picks up new users (SQL:
user.uidNumber is NULL
) and new projects (SQL:groups.gidNumber is NULL
) and assigns them ids.
sv_mailman
- queries themail_group_list
table, and creates/updates the mailing lists onlists.gnu.org
.- Source code: /backend/mail/sv_mailman.in.
- This script is tightly-coupled with the php code in /frontend/php/mail/admin/index.php.
- The
index.php
code sets themail_group_list.status
field to the requested new status, and thesv_mailman
script executes the changes and updates the database. sv_mailman
callsmgt0:/usr/sbin/newlist
. Thenewlist
script is a stub which delegates the call tolists.gnu.org
using SSH. See MailSystem for more details.
The gist ofmgt0:/usr/sbin/newlist
is:#!/bin/bash list_full_name=$1 admin_mail=$2 password=$3 cat <<EOF | ssh -T -i ~/.ssh/id_dsa.lists.gnu.org list@lists.gnu.org COMMAND=newlist LIST_FULL_NAME=$list_full_name ADMIN_MAIL=$admin_mail PASSWORD=$password EOF
sv_cleaner
- cleans up left-overs from interrupted php code?- source code: /backend/misc/sv_cleaner.in.
- FIXME: investigate this script.
sv_reminder
- sends email reminders to users.- source code: /backend/misc/sv_reminder.in.
The help message says:
This script will send reminder to users in some specific cases. Case 1: an item is about to start or finish, all people from the item notification list should be warned. Case 2: an user decided to receive a batch of open item assigned to him each x days, minimum priority being > 5 Case 3: a project admin decided that members assigned to some item of should receive a batch each x days, minimum priority > 5 (Case 4: a project admin decided that members assigned to some item whose statut/resolution havent changed for some period should receive a batch each x days)
FIXME: investigate this script.
www.gnu.org update mechanism
Mailing list configuration
Additional functionality
super-user and impersonate on frontend
frontend configuration files
(files which are not in the administration
(or any version control)
repositories)