FrontEnd Development Sites

With the new servers we aim to provide development copies of the frontend's PHP code, enabling savannah admins/volunteers to develop features and fix bugs more quickly.

Development websites will have URLs such as https://jsmith.frontend0.sv.gnu.org. The PHP code will be hosted in each developer's home directory, (e.g. frontend0:/home/jsmith/savannah/savane/).

See also: FrontEndSetup, SavannahHosts.

Step 1: DNS Configuration

Add a new DNS CNAME record, currently done on mgt0.savannah.gnu.org

See general DNS instructions in DNS.

A wildcard subdomain CNAME record was added to mgt0:/etc/bind/master/savannah.footer:

*.frontend0   CNAME   frontend0

The DNS setup on mgt0 ensures that for every subdomain (e.g. jsmith.frontend0) the hostname will be included in all four possibilities:

jsmith.frontend0.savannah.gnu.org
jsmith.frontend0.savannah.nongnu.org
jsmith.frontend0.sv.gnu.org
jsmith.frontend0.sv.nongnu.org

Step 2: MySQL database copy

Create a dedicated copy of 'savane' mysql database, to ensure changes on the development site do not affect the production site.

Create a new MySQL database and dedicated user (Example in internal0:/root/agn/create-savane-dev-db.sh. This script sets a password for the user). The result is:

mysql> select user,host from mysql.user where user='jsmith';
+---------+----------------------------+
| user    | host                       |
+---------+----------------------------+
| jsmith  | frontend.savannah.gnu.org  |
| jsmith  | frontend0.savannah.gnu.org |
| jsmith  | localhost                  |
+---------+----------------------------+
3 rows in set (0.01 sec)

Create a script to copy the existing data to the development database copy. See example in internal0:/root/agn/copy-savane-db-to-agn.sh. This process takes about 1 minute - safe to re-run whenever an update is needed. All existing data in the development DB will be deleted.

NOTE: An improved version is being discussed where only a smaller subset of the database is copied, to save disk space on the server.

Test connection from frontend0:

## Ensure it requires a password:
$ mysql -h internal0.sv.gnu.org -u jsmith savane_jsmith
ERROR 1045 (28000): Access denied for user 'jsmith'@'frontend0.savannah.gnu.org' (using password: NO)

## With the password:
$ mysql -h internal0.sv.gnu.org -u jsmith -p savane_jsmith
Enter password: ******
mysql>

## Ensure the dev user (jsmith) can not access the real database:
$ mysql -h internal0.sv.gnu.org -u jsmith -p savane
Enter password:
ERROR 1044 (42000): Access denied for user 'jsmith'@'frontend0.savannah.gnu.org' to database 'savane'

Step 3: Setup direcotries

The development site use dedicated directories, to ease development and troubleshooting.

cd /home/jsmith
mkdir savannah
cd savannah

# dev-site savane's php configuration
mkdir ./etc

# apache's access+error+php logs
mkdir -p ./var/logs

# uploads (for project registration)
mkdir -p ./var/www/submissions_uploads

# uploads (for trackers / attached files)
mkdir -p ./var/lib/savane/trackers_attachments

Ensure the directories are writable by www-data (run as root):

cd /home/jsmith/savannah/var
for i in ./logs \
         ./www/submissions_uploads \
         ./lib/savane/trackers_attachments ; do \
   chgrp www-data $i ; \
   chmod g+w $i ; \
done

Step 4: PHP frontend code

Clone the savane repository

cd /home/jsmith/savannah
git clone git://git.sv.gnu.org/administration/savane.git

The current PHP code (the same code running on the existing/old 'frontend') is few old PHP constructs. Few patches are required to make it run on modern PHP installations. The updated repository is available in frontend0:/home/agn/savannah/savane, or can be downloaded here: wget -O sv-local-run.patch.xz 'https://savannah.gnu.org/support/download.php?file_id=32220'

FIXME: push these to savane.git repository (if not in 'master', then at least in a separate branch).

Inside the dev. copy of savane, configure the copy for local installation (be sure to read below for warnings and notes about this 'installation'):

cd /home/jsmith/savannah/savane
./bootstrap
./configure --prefix=$HOME/savane-usr
make
make install

Notes:

1. The make step builds the Savane perl modules and scripts (called 'backend' - somewhat of a misnomer for the perl scripts running on 'frontend').

2. The 'make' step also prepares some images in <savane>/frontend/php/images from *.orig.png which are in the git repository (e.g. <savane>/frontend/php/images/Savane.theme/mist/trash.png generated from <savane>/frontend/php/images/Savane.theme/mist/trash.orig.png.) See <savane>/frontend/php/images/Makefile.am for the commands.

   Without this step, some image URLs will result in HTTP-404 errors.

3. The 'make install' will create the following directory structure:

   savane-usr
   |-- bin
   |-- etc
   |   `-- savane
   |       `-- content
   |           |-- account
   |           |-- admin
   |           |-- arch
   |           |-- bzr
   |           |-- cvs
   |           |-- git
   |           |-- hg
   |           |-- mail
   |           |-- my
   |           |-- people
   |           |-- predefined_mails
   |           |-- project
   |           |-- register
   |           `-- svn
   |-- lib
   |   `-- perl
   |       `-- 5.18.2
   |           `-- auto
   `-- share
       |-- locale
           |   |-- ca
   ...
   

4. make install does not install savane's PHP frontend code. The PHP code is served directory from the git repository directory. All make install does is to prepare the backend perl scripts and the content files. When using it for development (as opposed to the production website) it can be helpful to know which perl scripts are installed/needed.

5. The installed content directory ($HOME/savane-usr/content/etc/content in the example above) is not used for the development website. See below for manual configuration (which is just copied from the real website's /etc directory). But for future development, it could be useful to know which content files are installed/needed.

Step 5: Savannah's dev-site configuration files,content

The PHP code reads the configuration from a .savane.conf.php file, in a directory defined by SAVANE_CONF in the apache configuration file (see below).

For the development site, the files will be stored in /home/jsmith/savannah/etc/savane.

The configuration for the production site are in /etc/savane.

Copy the configuration files from the production website, and update ownership and permissions (as root):

cd /home/jsmith/savannah/
cp -r /etc/savane ./etc/
chown -R jsmith:jsmith ./etc
cd ./etc/savane
chgrp www-data .savane.conf.php .savane.pass nongnu-conf/.savane.conf.php
chmod o= .savane.conf.php .savane.pass nongnu-conf/.savane.conf.php

NOTE: At this point, the files contain the MySQL user/password for the production website - DO NOT FORGET TO CHANGE THEM. The following settings must be updated:

$ cd /home/jsmith/savannah/etc/savane
$ grep '_db' .savane.conf.php
$sys_dbhost="208.118.235.78";
$sys_dbname="savane_jsmith";
$sys_dbuser="jsmith";
$sys_dbpasswd=rtrim(file_get_contents(dirname(__FILE__).'/savane.pass'));

$ cat savane.pass
XXXXXX

$ cat savane.ini
[database]
HOST=208.118.235.78
USER=jsmith
PASSWORD=XXXXXXXX
NAME=savane_jsmith

The following settings should also be updated in .savane.conf.php:

$sys_default_domain="jsmith.frontend0.savannah.gnu.org";
$sys_brother_domain="jsmith.frontend0.savannah.nongnu.org";
$sys_debug_nobasehost=true; #disable gnu/nongnu redirection
$sys_incdir="/home/jsmith/savannah/etc/savane/content/gnu-content";
$sys_name="Savannah-jsmith-DEV";
$sys_upload_dir="/home/jsmith/savannah/var/www/submissions_uploads/";
$sys_appdatadir="/home/jsmith/savannah/var/lib/savane";
$sys_trackers_attachments_dir=$sys_appdatadir . "/trackers_attachments";
$sys_debug_email_override_address = "savannah-hackers-tests@gnu.org";

The following settings should be updated in ./nongnu-conf/.savane.conf.php (note that default-vs-brother are swapped in the nongnu file):

$sys_default_domain="agn.frontend0.savannah.nongnu.org";
$sys_brother_domain="agn.frontend0.savannah.gnu.org";
$sys_incdir="/home/agn/savannah/etc/savane/content/nongnu-content";

See FrontEndHostRedirection to learn about gnu/nongnu host redirection (keep the debug variable true unless you read and understand the redirection page).

See FrontEndNotificationEmails to learn about overriding emails with $sys_debug_email_override_address.

The outcome should look like:

$ cd /home/jsmith/savannah/etc/savane
$ ls -la
total 168
drwxr-xr-x 5 jsmith jsmith        4096 Sep  5 22:00 .
drwxrwxr-x 5 jsmith jsmith        4096 Jul 27 23:30 ..
-rwxr-x--- 1 jsmith www-data      1616 Sep  5 21:58 .savane.conf.php
-rw-r--r-- 1 jsmith jsmith      123540 Jul 27 23:30 cgitrepos
drwxr-xr-x 5 jsmith jsmith        4096 Jul 27 23:30 content
drwxr-xr-x 2 jsmith jsmith        4096 Jul 27 23:30 nongnu-conf
-rwxr-x--- 1 jsmith jsmith        8533 Sep  5 22:00 savane.conf.pl
-rw-r----- 1 jsmith jsmith          90 Sep  5 21:59 savane.ini
-rw-r----- 1 jsmith www-data        25 Sep  5 21:58 savane.pass
drwxr-xr-x 2 jsmith jsmith        4096 Jul 27 23:30 test-conf

Step 6: Apache configuraion

Create a separate apache configuration file: frontend0:/etc/apache2/sites-available/sv-jsmith-dev.conf A symlink should exist in: frontend0:/etc/apache2/sites-enabled/sv-jsmith-dev.conf (A good starting point is /etc/apache2/sites-available/sv-agn-dev.conf.)

Hostnames

The apache config file defines the following virtual host names

jsmith.frontend0.savannah.gnu.org
jsmith.frontend0.sv.gnu.org
jsmith.frontend0.savannah.nongnu.org
jsmith.frontend0.sv.nongnu.org

See FrontEndHostRedirection for the gory details of savannah's multiple hostnames configuration (e.g. gnu/nongnu redirections).

SSL Certificates

The apache configuration file includes an alias to certbot's ".well-known" directory to allow Let's-Encrypt SSL certificate generation. The following URLs will resolve to frontend0:/home/certbot/www/.well-known:

http://jsmith.frontend0.sv.gnu.org/.well-known/
http://jsmith.frontend0.sv.nongnu.org/.well-known/
http://jsmith.frontend0.savannah.gnu.org/.well-known/
http://jsmith.frontend0.savannah.nongnu.org/.well-known/

To verify the setup, try to access the following file:

wget http://jsmith.frontend0.savannah.gnu.org/.well-known/foo

Which corresponds to frontend0:/home/certboot/www/.well-known/foo. If the file is downloaded successfully, certbot's certificate registration/renwal process should 'just work'.

See https for more information about savannah's certbot/let's-encrypt certificates setup. See the section "Registering additional sub-domains" at the bottom of that page for example of adding a certificate for jsmith.frontend0.

After registering an SSL certificate for this new subdomain, use the following apache configuration statements:

<VirtualHost *:443>
  [... other SSL related settings ...]
  SSLCertificateFile      /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org/cert.pem
  SSLCertificateKeyFile   /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org/privkey.pem
  SSLCertificateChainFile /home/certbot/etc/live/jsmith.frontend0.savannah.gnu.org/chain.pem
</VirtualHost>

Document root

The PHP code is served from the savane git repository directory:

DocumentRoot /home/jsmith/savanne/savane/frontend/php

Savane's PHP configuration

In this apache config file, an environment variable SAVANE_CONF should point to the development site's configuration directory (e.g. frontend0:/home/jsmith/savannah/etc/savane).

SetEnv SAVANE_CONF /home/agn/savannah/etc/savane/ SetEnv SAVANE_CONF /home/agn/savannah/etc/savane/nongnu-conf/

Sanity checks when visiting https://jsmith.frontend0.savannah.gnu.org

1. Take note of the of the page (usually appears in the title of the web-browser or the open tab). It should say "Welcome [Savannah-jsmith-DEV]" which corresponds to the $sys_name variable in /home/jsmith/savannah/etc/savane/.savane.conf.php. If it says anything else, the PHP code for this development site is not reading the correct configuration file - check your apache configuration. If it says "Welcome [Savane]", it is likely reading the default configuration file from /etc/savane/.savane.conf.php.
2. If you see a red "error" box at the top of the page saying: 'Warning: Savane was not able to read "" site-specific information, please contact administrators' it likely means the $sys_incdir variable points to an invalid directory.

Password-protection

This is a development website, not meant for public consumption. Add a simple password, so the website is available only to the developer (e.g. when working on new, incomplete features).

<Location />
    # Require all granted
    AuthType Basic
    AuthName "Jane Smith's DEV Website"
    AuthUserFile /etc/apache2/sv-jsmith-dev-passwd
    Require valid-user
</Location>

Upload directory

Update the upload directory alias with:

Alias /submissions_uploads /home/jsmith/savannah/var/submissions_uploads/
<Directory "/home/jsmith/savannah/var/submissions_uploads/">
  [...]
</Directory>

Apache logs

Update the location of the log files to write them to the developer's directory:

# Apache Logs for development site
CustomLog /home/jsmith/savannah/var/logs/agn.access.log combined
ErrorLog /home/jsmith/savannah/var/logs/agn.error.log
php_admin_value error_log /home/jsmith/savannah/var/logs/agn.php-error.log