Mirroring at Savannah
NOTE: This is an updated version of the previous Mirmon page.
Savannah offers two public-facing mirroring services:
ftpmirror.gnu.org- redirects to world-wide mirrors offtp.gnu.org. These mirrors carry official releases of GNU pacakges. This page refers to it asgnu mirror.download.savannah.{nongnu,gnu}.org- redirects to world-wide mirrors of Savannah download area. Typically used for non-GNU software, but GNU packages can store files there as well. This page refers to it asnongnu mirror.
Adding mirrors
Occasionally, a new mirror is submitted. Usually the webmasters inform us, but people may come to us directly.
When we are told of a new mirror, we verify it, then
- On mgt1, add contact information to
/root/administration/mirrors-contacts.txt; - On download0.savannah.gnu.org, add the mirror URLs to
/srv/download/00_MIRRORS.html(alphabetical, check list online) and/srv/download/00_MIRRORS.txt(no order). Supported protocols are FTP, HTTP, HTTPS, RSYNC.
Client-side overview
A user requesting a download from ftpmirror.gnu.org will receive
an HTTP/302 response redirecting to a mirror closest to the user's
originating IP:
$ curl -o /dev/null -D /dev/stdout --silent http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
HTTP/1.1 302 Found
Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
[...]
A user requested a download from download.savannah.gnu.org will be
redirected to an intermediate redirection URL, and then to the mirror:
$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases/test-project/README
HTTP/1.1 302 Found
Location: http://download.savannah.gnu.org/releases-redirect/test-project/README
[...]
$ curl -o /dev/null -D /dev/stdout --silent http://download.savannah.gnu.org/releases-redirect/test-project/README
HTTP/1.1 302 Found
Location: http://nongnu.askapache.com//test-project/README
[...]
The technical reason for this extra redirection is that
download.savannah.gnu.org serves directory listing directly, and
only redirects actual files. See nginx details below.
Server-side redirection overview
Redirection is implemented using a Perl CGI script
called from nginx running on download0.sv.gnu.org.
Both gnu and nongnu redirections are served from
download0.savannah.gnu.org VM. ftpmirror.gnu.org is
CNAME'd to ftpmirror.sv.gnu.org, which points
to the same IP as download0 (remember Savannah admins
only control *.sv.gnu.org DNS entries).
The nginx configuration files are:
download0:/etc/nginx/sites-available/ftpmirror,
download0:/etc/nginx/sites-available/ftpmirror-common.inc,
download0:/etc/nginx/sites-available/ftpmirror-common-https.inc,
download0:/etc/nginx/sites-available/download,
download0:/etc/nginx/sites-available/download-common.inc,
download0:/etc/nginx/sites-available/download-common-https.inc.
The *-common*.inc files contain the CGI configurations.
The other files include them twice (once for HTTP, once for HTTPS).
The CGI configuration for gnu mirror is:
location / {
gzip off;
# In this case, the ENTIRE path is passed on the to cgi script, e.g.
# given http://ftpmirror.sv.gnu.org/datamash/datamash-1.1.0.tar.gz we pass
# '/datamash/datamash-1.1.0.tar.gz' to the CGI script.
# So there is no need to use 'fastcgi_split_path_info' command.
fastcgi_param PATH_INFO $uri;
fastcgi_param LOG_DIR /opt/savannah/stat/ftp/log;
fastcgi_param MIRROR_FILE $mirror_file;
fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
The CGI configuration for nongnu mirror is:
# Redirect to the CGI if it's a file URL.
rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;
# Directory listing if it's a directory.
location /releases {
alias /srv/download;
fancyindex on; # Enable fancy indexes.
}
# Actual redirection mechanism.
location /releases-redirect {
gzip off;
# This command tells NGINX how to extract the path info part
# from the requested URL: it trims the 'releases-redirect' part,
# leaving only what follows it.
fastcgi_split_path_info ^(/releases-redirect)(/?.+)$;
# This command takes NGINX's path_info variable (extracted above),
# and sends it to the FastCGI daemon as the PATH_INFO environment variable.
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param LOG_DIR /opt/savannah/stat/dl/log;
fastcgi_param MIRROR_FILE $mirror_file;
fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect;
fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat;
fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat;
include fastcgi_params;
fastcgi_pass unix:/var/run/fcgiwrap.socket;
}
The $mirror_file variable is set to one of
/opt/savannah/mirrors/active-mirror-lists/{gnu,download}{,-https}.txt
depending on whether it's served via HTTP or HTTPS and whether it's
used for gnu mirror or nongnu mirror.
The mirror list text files (gnu.txt and download.txt)
contain an auto-generated list of active mirrors and region redirections:
# head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
http://us.mirrors.cicku.me/gnu/ us
http://mirror.freedif.org/GNU/ sg
http://fosszone.csd.auth.gr/gnu/ gr
http://mirror.marwan.ma/gnu/ ma
http://ftp.task.gda.pl/pub/gnu/ pl
# tail -n5 /opt/savannah/mirrors/active-mirror-lists/gnu.txt
ye az
yt za
za za
zm za
zw za
(Yemen [YE] uses the mirrors located in Azerbaijan [AZ], Zimbabwe [ZW] uses the mirrors located in South Africa [ZA].)
See 'mirmon' section below to learn how these files are generated.
The Perl script mirror-redirect uses Geo::IP module to detect users' regions based on their IP.
Server-side Mirmon overview
The authoritative lists of mirrors are managed manually. For
gnu mirror, GNU webmasters update the lists (http://www.gnu.org/prep/ftp.html, generated from http://www.gnu.org/prep/FTP). Fornongnu mirror, Savannah admins maintain the lists as explained above.mirmon is used to check each mirror and determine how up-to-date it is.
We maintain 3 separate mirmon configurations:
gnu- list of HTTP(S)-only mirrors for ftpmirror.gnu.org. Thegnulist is used later to compile the list of active mirrors for the redirection CGI script.allgnu- list of HTTP(S), FTP, RSYNC mirrors for ftpmirror.gnu.org. Theallgnulist is used only by human admins to check which mirrors are up-to-date.nongnu- list of mirrors for download.savannah.gnu.org. Thenongnulist is used later to compile the list of active mirrors for the redirection CGI script.
The mirmon input lists (of mirrors to check) are here:
The lists are stored here:
$ ls -lhog /opt/savannah/mirrors/mirmon-lists/ -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt lrwxrwxrwx 1 28 Feb 20 23:28 nongnu-mirror-list.txt -> /srv/download/00_MIRRORS.txtThe script sv_gnu-mirmon fetches the gnu list and saves it locally in the two versions (
gnuandallgnu). Thenongnulist is symlinked to the manually-managed file.These files are the input to mirmon.
The three mirmon configuration files are:
$ ls -1 /etc/mirmon/ mirmon-allgnu.conf mirmon-gnu.conf mirmon-nongnu.confWhen
mirmonis executed (once for each configuration), it reads the corresponding list file, and generates an HTML report page and a 'state' textual file:/opt/savannah/www/mirmon/gnu/index.html /opt/savannah/www/mirmon/nongnu/index.html /opt/savannah/www/mirmon/allgnu/index.html # ls -lhog /var/lib/mirmon/state-* -rw-r--r-- 1 16K Feb 21 19:42 /var/lib/mirmon/state-allgnu -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnuA 'state' file contains the status for each mirror:
$ head -n5 /var/lib/mirmon/state-gnu http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.gnu.org/ 1707624961 ok 1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721 http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1707617761 ok 1707654121 ssssssssssssss 1707626399-ssssssssssssss 1707654121 http://ftp.cc.uoc.gr/mirrors/gnu/ 1707581761 ok 1707654121 ssssssssssssss 1707626399-ssssssssssssss 1707654121 http://ftp.eq.uc.pt/software/unix/gnu/ 1707621361 ok 1707657721 ssssssssssssfs 1707628921-ssssssssssssss 1707657721 http://ftp.fau.de/gnu/ 1707646561 ok 1707657721 ssssssssssssss 1707632521-ssssssssssssss 1707657721Two auxiliary scripts
/opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.shtrivially invoke sv_mirmon-to-geoip; that script reads the state file and the mirror list and generates simple text files containing the active and up-to-date mirrors and region redirections:# ls /opt/savannah/mirrors/active-mirror-lists/ download-https.txt download.txt gnu-https.txt gnu.txtThese are the four files used in the CGI scripts (see above section).
The 3 mirmon invocations are executed as cronjobs in
/etc/cron.d/sv-mirmon:# Update the authoritative list of GNU ftp mirrors (mirroring ftp.gnu.org). 3 1 * * * root sv_gnu-mirmon --http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt 6 1 * * * root sv_gnu-mirmon --all /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt # Check the status of each mirror (and regenarate the HTML report page). # For gnu/nongnu - also update the list of active mirrors (for the mirror CGIs). 22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get update >/var/log/mirmon/gnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh 32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q -get update >/var/log/mirmon/nongnu 2>&1 && /opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh 42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q -get update >/var/log/mirmon/allgnu 2>&1
Diagnostics and Troubleshooting
The three mirmon HTML reports are viewable here: http://download.savannah.gnu.org/mirmon/.
An additional page is used for testing: http://download.savannah.gnu.org/mirror-check/, http://ftpmirror.savannah.gnu.org/mirror-check/, https://download.savannah.gnu.org/mirror-check/, https://ftpmirror.savannah.gnu.org/mirror-check/.
The NGINX configurations are
*-common.inc:location /mirror-check/ { gzip off; fastcgi_split_path_info ^(/mirror-check)(/?.+)$; fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_param MIRROR_FILE $mirror_file; fastcgi_param SCRIPT_FILENAME /opt/savane/bin/mirror-redirect; fastcgi_param GEOIP_DB /usr/share/GeoIP/GeoIP.dat; fastcgi_param GEOIP6_DB /usr/share/GeoIP/GeoIPv6.dat; fastcgi_param TEST 1; include fastcgi_params; fastcgi_pass unix:/var/run/fcgiwrap.socket; }The web page accepts two optional parameters,
- addr overrides the detected IP address
- cc overrides the detected country code
For example,
$ wget -o /dev/null -O - \ 'https://ftpmirror.gnu.org/mirror-check/?addr=209.51.188.113&cc=lt' GNU Savannah - mirror check Requested path: / Query string: addr=209.51.188.113&cc=lt Detected remote address: 209.51.188.113 (overridden in query string) Detected country: United States Country code: lt (overridden in query string) Suggested mirror: https://sunsite.icm.edu.pl/pub/gnu All mirrors from pl: https://ftp.task.gda.pl/pub/gnu/ https://ftp.man.poznan.pl/gnu/ https://sunsite.icm.edu.pl/pub/gnu/Mirmon cronjob logs are saved in
/var/log/mirmon. Current implementation saves only the most recent run.Mirmon state files should be generated at 22/32/42 minutes of each hour (defined in the cron file):
# ls -lhog /var/lib/mirmon/ total 84K -rw-r--r-- 1 27K Feb 11 09:17 debian-mirror-check.html -rw-r--r-- 1 3.9K Feb 11 09:17 state -rw-r--r-- 1 24K Feb 11 08:42 state-allgnu -rw-r--r-- 1 18K Feb 11 08:22 state-gnu -rw-r--r-- 1 4.2K Feb 11 08:32 state-nongnuThe active mirror list files should have similar timestamp:
# ls -lhog /opt/savannah/mirrors/active-mirror-lists/ total 24K -rw-r--r-- 1 1.7K Feb 11 08:32 download-https.txt -rw-r--r-- 1 2.1K Feb 11 08:32 download.txt -rw-r--r-- 1 4.4K Feb 11 08:22 gnu-https.txt -rw-r--r-- 1 4.2K Feb 11 08:22 gnu.txtRedirection script statistics are available at https://download.savannah.gnu.org/dl-stats/ (for download) and https://download.savannah.gnu.org/ftp-stats/ (for ftpmirror).
Other summaries based on latest nginx logs are available (for ftpmirror only) at https://download.savannah.gnu.org/ftpmirror-stats/.