Monday, 20 Oct, 2008

debian logo

debian logo

Someday at work, it was decided that having a local debian mirror would be beneficial.

After some research, I choose total flexibility, by adopting the anonftpsync shell script.

Anonftpsync is a self-descriptive-log-and-mail kind of script.

You'll just have to customize it a bit in order to be able to use it.

For convenience, below you'll find the variations that I've used (normally it's just commented, so some parts have been uncommented out in order to enable them):

#! /bin/sh set -e  
# This script originates from http://www.debian.org/mirror/anonftpsync  
# modified by Andrea Matesi  
# CVS: cvs.debian.org:/cvs/webwml - webwml/english/mirror/anonftpsync 
# Version: $Id: anonftpsync,v 1.30 2007/09/06 18:05:44 joy Exp $  
# Note: You MUST have rsync 2.6.4 or newer, which is available in sarge 
# and all newer Debian releases, or at http://rsync.samba.org/  
# Don't forget: 
# chmod u+x anonftpsync  
# Set the variables below to fit your site. You can then use cron to have 
# this script run daily to automatically update your copy of the archive.  
# TO is the destination for the base of the Debian mirror directory 
# (the dir that holds dists/ and ls-lR). 
# (mandatory)  TO=/mnt/backup/mirror  
# RSYNC_HOST is the site you have chosen from the mirrors file. 
# (http://www.debian.org/mirror/list-full) 
# (mandatory)  RSYNC_HOST=debian.fastweb.it  
# RSYNC_DIR is the directory given in the "Packages over rsync:" line of 
# the mirrors file for the site you have chosen to mirror. 
# (mandatory)  RSYNC_DIR=debian/  
# LOGDIR is the directory where the logs will be written to 
# (mandatory)  LOGDIR=/var/log  
# ARCH_EXCLUDE can be used to exclude a complete architecture from 
# mirrorring. Please use as space seperated list. 
# Possible values are: 
# alpha, amd64, arm, hppa, hurd-i386, i386, ia64, m68k, mipsel, mips, powerpc, s390, sh and sparc 
# 
# There is one special value: source 
# This is not an architecture but will exclude all source code in /pool 
# 
# eg. ARCH_EXCLUDE="alpha amd64 arm hppa hurd-i386 ia64 m68k mipsel mips powerpc s390 sh sparc" 
# 
# With a blank ARCH_EXCLUDE you will mirror all available architectures 
# (optional)  #ARCH_EXCLUDE=  
# EXCLUDE is a list of parameters listing patterns that rsync will exclude, in 
# addition to the architectures excluded by ARCH_EXCLUDE. 
# 
# Use ARCH_EXCLUDE to exclude specific architectures or all sources 
# 
# --exclude stable, testing, unstable options DON'T remove the packages of 
# the given distribution. If you want do so, use debmirror instead. 
# 
# The following example would exclude mostly everything: EXCLUDE="\ --exclude stable/ --exclude testing/
 --exclude unstable/ \ --exclude source/ \ --exclude *.orig.tar.gz --exclude *.diff.gz --exclude *.dsc 
\ --exclude /contrib/ --exclude /non-free/ \ "  
# With a blank EXCLUDE you will mirror the entire archive, except the 
# architectures excluded by ARCH_EXCLUDE. 
# (optional)  #EXCLUDE=  
# MAILTO is the address to send logfiles to; 
# if it is not defined, no mail will be sent 
# (optional)  [email protected]  
# There should be no need to edit anything below this point, unless there 
# are problems.  
#-----------------------------------------------------------------------------
#  
# If you are accessing a rsync server/module which is password-protected, 
# uncomment the following lines (and edit the other file).  
# . ftpsync.conf 
# export RSYNC_PASSWORD 
# RSYNC_HOST=$RSYNC_USER@$RSYNC_HOST  
#-----------------------------------------------------------------------------
#  
# Check for some environment variables if [ -z $TO ] || [ -z $RSYNC_HOST ] || [ -z $RSYNC_DIR ] || [ -z $LOGDIR ]; 
then echo "One of the following variables seems to be empty:" echo "TO, RSYNC_HOST, RSYNC_DIR or LOGDIR" exit 2 fi

if ! [ -d ${TO}/project/trace/ ]; then 
# we are running mirror script for the first time umask 002 mkdir -p ${TO}/project/trace fi  
# Note: on some non-Debian systems, hostname doesn't accept -f option. 
# If that's the case on your system, make sure hostname prints the full 
# hostname, and remove the -f option. If there's no hostname command, 
# explicitly replace `hostname -f` with the hostname.  HOSTNAME=`hostname`  
# The hostname must match the "Site" field written in the list of mirrors. 
# If hostname doesn't returns the correct value, fill and uncomment below 
# HOSTNAME=mirror.domain.tld  LOCK="${TO}/Archive-Update-in-Progress-${HOSTNAME}"  
# The temp directory used by rsync --delay-updates is not 
# world-readable remotely. It must be excluded to avoid errors. TMP_EXCLUDE="--exclude .~tmp~/"  
# Exclude architectures defined in $ARCH_EXCLUDE for ARCH in $ARCH_EXCLUDE; do EXCLUDE=$EXCLUDE"\ --exclude binary-$ARCH/
\ --exclude disks-$ARCH/ \ --exclude installer-$ARCH/ \ --exclude Contents-$ARCH.gz \ --exclude Contents-$ARCH.diff/ 
\ --exclude *_$ARCH.deb \ --exclude *_$ARCH.udeb " if [ "$ARCH" == "source" ]; then SOURCE_EXCLUDE="\ --exclude *.tar.gz 
\ --exclude *.diff.gz \ --exclude *.dsc " fi done  
# Logfile LOGFILE=$LOGDIR/debian-mirror.log  
# Get in the right directory and set the umask to be group writable 
# cd $HOME umask 002  
# Check to see if another sync is in progress if [ -f "$LOCK" ]; then if [ "`find $LOCK -maxdepth 1 -amin -360`" = "" ]; then 
# Note: this requires the procps ps; for other ps', adjust as necessary if ps ax | grep '[r]'sync | grep -q $RSYNC_HOST; 
then echo "stale lock found, but a rsync is still running, aiee!" exit 1 
else echo "stale lock found (not accessed in the last 6 hours), forcing update!" rm -f $LOCK fi 
else echo "current lock file exists, unable to start rsync!" exit 1 fi fi  touch $LOCK 
# Note: on some non-Debian systems, trap doesn't accept "exit" as signal # specification. 
If that's the case on your system, try using "0". trap "rm -f $LOCK" exit  set +e  
# First sync /pool rsync --recursive --links --hard-links --times --verbose \ $TMP_EXCLUDE $EXCLUDE $SOURCE_EXCLUDE \ 
$RSYNC_HOST::$RSYNC_DIR/pool/ $TO/pool/ >> $LOGFILE 2>&1 result=$?  if [ 0 = $result ]; then 
# Now sync the remaining stuff rsync --recursive --links --hard-links --times --verbose --delay-updates --delete-after \
 --exclude "Archive-Update-in-Progress-${HOSTNAME}" \ --exclude "project/trace/${HOSTNAME}"
 \ $TMP_EXCLUDE $EXCLUDE $SOURCE_EXCLUDE \ $RSYNC_HOST::$RSYNC_DIR $TO >> $LOGFILE 2>&1  LANG=C 
date -u > "${TO}/project/trace/${HOSTNAME}" else echo "ERROR: Help, something weird happened" | tee -a $LOGFILE 
echo "mirroring /pool exited with exitcode" $result | tee -a $LOGFILE fi  if ! [ -z $MAILTO ]; 
then mail -s "debian archive synced" $MAILTO < $LOGFILE fi  savelog $LOGFILE >/dev/null  rm $LOCK

This script was then placed within crontab (in order to have it executed automatically).

Apache http was then configured for file-serving purposes (ie. by using a different virtual host).

Once all was setup and in place, the next time I had to manually install a debian distribution on a local box, I simply pointed the installer to grab the deb packages off my internal (local) repo and everything just worked as expected (including the speed of execution, since it was inside our LAN!).

Rate this post