Mirroring at Savannah
NOTE: This is an updated version of the previous Mirmon page.
Savannah offers two public-facing mirroring services:
ftpmirror.gnu.org
- redirects to world-wide mirrors offtp.gnu.org
. These mirrors carry official releases of GNU pacakges.ftpmirror.gnu.org
andgnu mirror
are used interchangeably in this page.download.savannah.{nongnu,gnu}.org
- redirects to world-wide mirrors of Savannah download area. Typically used for non-GNU software, but GNU packages can store files there as well.download
andnongnu mirror
are used interchangeably in this page.
Client-side overview
A user requesting a download from ftpmirror.gnu.org
will receive
an HTTP/302 response redirecting to a mirror closest to the user's
originating IP:
$ curl -o /dev/null -D /dev/stdout --silent http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
HTTP/1.1 302 Found
Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
[...]
A user requested a download from download.savannah.gnu.org
will be
redirected to an intermediate redirection URL, and then to the mirror:
$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases/test-project/README
HTTP/1.1 302 Found
Location: http://download.savannah.gnu.org/releases-redirect/test-project/README
[...]
$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases-redirect/test-project/README
HTTP/1.1 302 Found
Location: http://nongnu.askapache.com//test-project/README
[...]
The technical reason for this extra redirection is that
download.savannah.gnu.org
serves directory listing directly, and
only redirects actual files. See nginx details below.
Server-side redirection overview
Redirection is implemented using a perl cgi script,
called from nginx/fcgiwrap running on download0.sv.gnu.org
.
Both gnu and nongnu redirections are served from
download0.savannah.gnu.org VM. ftpmirror.gnu.org
is
CNAME'd to ftpmirror.sv.gnu.org
, which points
to the same IP as download0 (remember Savannah admin's
only control *.sv.gnu.org DNS entries).
The nginx configuration files are:
download0:/etc/nginx/sites-available/ftpmirror
,
download0:/etc/nginx/sites-available/ftpmirror-common.inc
,
download0:/etc/nginx/sites-available/ftpmirror-common-https.inc
,
download0:/etc/nginx/sites-available/download
,
download0:/etc/nginx/sites-available/download-common.inc
,
download0:/etc/nginx/sites-available/download-common-https.inc
.
The *-common*.inc
files contain the cgi configurations.
The other files include them twice (once for http, once for https).
The https CGI configuration for gnu/ftpmirror
is:
location / {
fastcgi_param PATH_INFO $uri;
fastcgi_param MIRROR_FILE /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt;
fastcgi_param SCRIPT_FILENAME /opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
The CGI configuration for nongun/download
is:
# redirect to the CGI if it's a file URL
rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;
# directory listing if it's a directory
location /releases {
alias /srv/download;
fancyindex on; # Enable fancy indexes.
}
# actual redirection mechanism
location /releases-redirect {
fastcgi_split_path_info ^(/releases-redirect)(/?.+)$;
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param MIRROR_FILE /opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
fastcgi_param SCRIPT_FILENAME /opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
The https versions substitute mirrors.txt with mirrors-https.txt
The mirror list text files (gnu-ftpmirror
and download-mirrors.txt
)
contain an auto-generated list of active mirrors:
# head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt
http://mirror.tochlab.net/pub/gnu/ ru
http://mirror.cedia.org.ec/gnu/ ec
http://www.gutscheinrausch.de/mirror/gnu/ de
http://www.artfiles.org/gnu.org/ de
http://gnu.mirrors.linux.ro/ ro
See 'mirmon' section below to learn how these files are generated.
The perl script mirror-redirect.pl
uses
Geo::IP and Geo::Mirror
modules
to read the mirror file, detect the closest region based on the user's
IP, and return an HTTP redirect to that mirror.
Server-side Mirmon overview
NOTE: Most of this section was copied from the Mirmon page and updated as needed.
The authoratative lists of gnu/nongnu mirrors are managed manually. For gnu/ftpmirrors, FSF admins or GNU webmasters update the lists (http://www.gnu.org/prep/ftp.html, generated from http://www.gnu.org/prep/FTP). For nongnu/download mirrors, Savannah admins update this file (and html version).
mirmon is used to checks each mirror and determine how up to date it is.
We maintain 3 separate mirmon configurations:
gnu
- list of HTTP-only mirrors for ftpmirror.gnu.org. Thegnu
list is used later by the redirection cgi script.allgnu
- list of HTTP,FTP,RSYNC mirrors for ftpmirror.gnu.org. Theallgnu
list is used only by human admins to check which mirrors are up-to-date.nongnu
- list of HTTP-only mirrors for download.savannah.gnu.org. Thenongnu
list is used later by the redirection cgi script.
The mirmon input lists (of mirrors to check) are here:
The lists are stored here:
$ ls -lhog /opt/savannah/mirrors/mirmon-lists/ -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt lrwxrwxrwx 1 28 Feb 20 23:28 nongnu-mirror-list.txt -> /srv/download/00_MIRRORS.txt
The script
/opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl
fetches the gnu/ftpmirror list and saves it locally in the two versions (gnu
andallgnu
). Thenongnu
list is symlinked to the manually-managed file.These files are the input to mirmon.
The three mirmon configuration files are:
$ ls -1 /etc/mirmon/ mirmon-allgnu.conf mirmon-gnu.conf mirmon-nongnu.conf
When
mirmon
is executed (once for each configuration), it reads the corresponding list file, and generates an HTML report page and a 'state' textual file:/opt/savannah/www/mirmon/gnu/index.html /opt/savannah/www/mirmon/nongnu/index.html /opt/savannah/www/mirmon/allgnu/index.html # ls -lhog /var/lib/mirmon/state-* -rw-r--r-- 1 16K Feb 21 19:42 /var/lib/mirmon/state-allgnu -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnu
A 'state' file contains the status for each mirror:
$ head -n5 /var/lib/mirmon/state-gnu http://ca.mirror.babylon.network/gnu/ 1487646961 ok 1487651999 s 1487651999-s 1487651999 http://download.nust.na/pub/ftp.gnu.org/gnu/ 1487643361 ok 1487651999 s 1487651999-s 1487651999 http://fr.mirror.babylon.network/gnu/ 1487643361 ok 1487651999 s 1487651999-s 1487651999 http://ftp-gnu-org.ip-connect.vn.ua/ 1487625361 ok 1487651999 s 1487651999-s 1487651999 http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1487556961 ok 1487651999 s 1487651999-s 1487651999
Two auxiliary scripts
/opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh
read the gnu/nongnu state files, and generate a simple text files containing only the active and up-to-date mirrors:$ ls -1 /opt/savannah/mirrors/active-mirror-lists/ download-mirrors-https.txt download-mirrors.txt gnu-ftpmirrors-https.txt gnu-ftpmirrors.txt
These are the four files used in the nginx/cgi scripts (see above section).
The 3 mirmon invocations are executed as cronjobs in
/etc/cron.d/sv-mirmon
:# Update the authoratative list of GNU ftp mirrors (mirroring ftp.gnu.org). 3 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl --http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt 6 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl --all /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt # Check the status of each mirror (and regenarate the HTML report page). # For gnu/nongnu - also update the list of active mirrors (for the mirror CGIs). 22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get update >/var/log/mirmon/gnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh 32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q -get update >/var/log/mirmon/nongnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh 42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q -get update >/var/log/mirmon/allgnu 2>&1
Diagnostics and Troubleshooting
The three mirmon HTML reports are viewable here: http://download0.savannah.gnu.org/mirmon/.
An additional CGI script exercise the Geo::IP module and the generated files: http://download0.savannah.gnu.org/mirror-check/ and http://ftpmirror0.savannah.gnu.org/mirror-check/.
The script is
/opt/savannah/cgi-bin/mirrors/mirror-check.pl
.The NGINX configurations are
ftpmirror-common.inc
anddownload-common.inc
:location /mirror-check/ { fastcgi_param GNU_MIRROR_FILE /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt; fastcgi_param NONGNU_MIRROR_FILE /opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt; fastcgi_param SCRIPT_FILENAME /opt/savannah/cgi-bin/mirrors/mirror-check.pl; include fastcgi_params; fastcgi_pass unix:/var/run/fcgiwrap.socket; }
Mirmon cronjob logs are saved in
/var/log/mirmon
. Current implementation saves only the most recent run.Mirmon state files should be generated at 22/32/42 minutes of each hour (defined in the cron file):
# ls -lhog /var/lib/mirmon/ total 60K -rw-r--r-- 1 7.3K Feb 21 20:22 state-gnu -rw-r--r-- 1 3.5K Feb 21 20:32 state-nongnu -rw-r--r-- 1 16K Feb 21 20:42 state-allgnu
The active mirror list files should have similar timestamp:
# ls -lhog /opt/savannah/mirrors/active-mirror-lists/ total 16K -rw-r--r-- 1 415 Oct 15 16:32 download-mirrors-https.txt -rw-r--r-- 1 1.1K Oct 15 16:32 download-mirrors.txt -rw-r--r-- 1 626 Oct 15 16:22 gnu-ftpmirrors-https.txt -rw-r--r-- 1 2.8K Oct 15 16:22 gnu-ftpmirrors.txt