Why clocks don't work in PNG

One of the good things, or perhaps the worst thing (it depends on where
and when) about my job is I travel a bit.  One of the more interesting
places (again in both good and bad ways) was Port Moresby,
Papua New Guinea. Lots of stuff doesn’t work too well over there. It’s
hard to get experience maintenance staff, its hard to get the right
parts over there and probably the hardest part is paying for it. 

I
was always surprised the clocks never work right. They are either
horribly fast or slow.  At first I thought the hotel had just not set
it right, so I set it to the right time. But the next morning it was
out again. What the hell was going on?

Most alarm clocks work
off the power. In both Australia and PNG this is 50Hz, so the clock
rectifies the AC power coming in to get 50 pulses a second. Count 50
pulses and you have a second, do this 60 times and tick over a minute.

We
take for granted that the frquency of power is right on 50Hz, but
talking with the students who were doing power engineering it is
actually quite hard to do. It’s like having a car with the same revs no
matter if you are going up or down a hill.

Anyhow, if you need
to wake up the right time in PNG, use the alarm clock on your phone.
Otherwise you may be jumping out of bed at 5am thinking you are running
late, then wondering where the sun went.

Changing Sites

While I originally had a blog site on Avogato, I just didnt seem to use it much.  I needed some place that I could put some writing that wasn’t quite up to a new whole page with the associated heavy work to format it but it had to go somewhere.

So I’ll try to put those middle entries into this place.

Now all I got to work out is how to link the thing to Planet Debian and I’ll be set.

Connecting to Internode IPv6 on Debian

If you are running Debian and are connected to the internet by Australian ISP Internode you can connect to their tunnel broker. This page describes how to do it with a few simple steps.

For more information about what Internode is doing with IPv6, have a quick look at the Internode IPv6 page. That page will give you some basic overall view of how the system is setup. Don’t use the instructions they give you. While they do work, its a lot more complicated their way.

##Information to collect

Before you start, you will need to know the following information:

* Your internode username and password, this is the same details that you put in your ADSL modem to connect to the ISP.
* Decide if you just want the Debian computer using IPv6 in “host mode” or you want everyone on your LAN to route through this computer in “router mode”.

##Installation
You will first need the gateway client program, which is found in the Debian package [gogoc](http://packages.debian.org/gogoc). If you are running in “router mode” you will also need to install
[radvd](http://packages.debian.org/radvd). Both of these packages are in the Debian main distribution so you can download them the normal way you get your Debian packages.

Edit the gogoc configuration file */etc/gogoc/gogoc.conf* to suit your situation, the important lines are:

userid=MY_USERNAME
passwd=MY_PASSWORD
server=sixgw.internode.on.net
auth_method=any
host_type=MY_HOST_TYPE

For MY_HOST_TYPE, use “host” or “router” depending if you want just this computer or everyone on your LAN to have IPv6 respectively.

##Starting gogoc for the first time
When you first start gogoc it will try to make a secured connection to the tunnel broker. The problem is that it needs to check the key you get is ok. This means that the first time you run gogoc you need to do it on the command line, like this:

server# invoke-rc.d gw6c stop
Stopping Gateway6 Client: gw6c.
server# gw6c
sixgw.internode.on.net is an unknown host, do you want to add its key?? (Y/N)
server# killall gw6c
server# invoke-rc.d gw6c start
Starting Gateway6 Client : gw6c.

The server key is now stored in */var/lib/gogoc/gogockeys.pub* and the program will start automatically with no further key problems.

##Checking its working
There are a few ways of checking your configuration is working:

* **pgrep gw6c** returns the pid of the program
* Use ifconfig program on interface tun0 or (if you are in router mode) eth0 should show inet6 addresses starting with 2001:44b8:: prefix which belongs to Internode.
* Browse to and watch the bouncing Google words.
* ifconfig output should look something like the following:

server$ /sbin/ifconfig  | egrep '(Link|inet6)'
eth0    Link encap:Ethernet  HWaddr 12:34:56:78:9a:bc
     inet6 addr: 2001:44b8:42:22::1/64 Scope:Global
     inet6 addr: fe80::234:56ff:fe78:9abc/64 Scope:Link
lo    Link encap:Local Loopback
     inet6 addr: ::1/128 Scope:Host
tun   Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
     inet6 addr: 2001:44b8:41::43/128 Scope:Global

Using Amanda for Backups

[AMANDA](http://www.amanda.org/) as the name says, is an advanced disk archiver or, more simply, a backup program. It’s a really useful program, let down by something approaching the worst documentation out there. I’ve had about 4 attempts on and off for about the same number of years at using it and deciphering the documentation. Finally I got it to work, but it should not be that hard. Who knows, maybe in
another 4 years I won’t hate Wiki documentation? Naah.

This document shows you what files you need to create, their format and how to use some of the programs. It’s probably only a brief get you started type of document, but at least you should be able to start. I have tested this setup on a bunch of [Debian](http://www.debian.org/) GNU/Linux servers but it should be reasonably similar for other systems.

Speaking of my systems, what I’m writing here works for me with my setup. It’s quite likely it will work for you, but you should verify and test everything. If something is not quite right and you lose all your valuable
files because something is wrong in here, you should of tested it properly yourself. In any case, a non-tested backup system is almost as useless as a non-working one.

#Determine your backup regime and other details
The first step is to decide what you want to backup. Do you want to do everything? Just home files? The variety is quite large and it is just a matter of working out how to tell AMANDA how to do it. AMANDA works
in chunks of partitions, so if you have everything in one partition as / then that’s one big chunk. All is not lost however as you can use some tricks to exclude some type of files or directories.

For my setup, I backup all of /, most of /var except logs and cache and /home with the exception of some junk. The method for backing up all of a partition can be done differently than backing up part of it. I’m not
sure, but I think its quicker to backup all of a partition when you can.

Next, think of a name to call the backup regime. I call mine “normal” and thats what I’ll use here. AMANDA calls the name “the config” in the manual pages. It’s just a label for the set of configuration files.

#Determining tapetype

The configuration file (see next section) will need to know the tapetype. If you don’t know what drive model you have, you can often get some information about the device in the **/proc/scsi/scsi** file. My file shows my ancient Sony tape drive.

Attached devices:
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: SONY     Model: SDT-5000&
nbsp;        Rev: 3.30
  Type:   Sequential-Access    &nbs
p;           ANSI SC
SI revision: 02

The tapetype is a bunch of parameters about your tape drive and they are different for
every device. Have a look at [AMANDA tapetype list](http://amanda.sourceforge.net/fom-serve/cache/45.html) to see if yours is there.

Failing that, use the **amtapetype** program. Be warned, it will take a long time for this program to work. When I mean a long time, I’m talking about 2 hours or so. That’s roughly how long it took for me.

To run it, type:

server# amtapetype /dev/nst0

and then wait, and wait… Eventually you’ll get something that you can cut and paste into your amanda.conf /dev/st0 is the device the tapedrive uses, try that one first if you have a SCSI tape drive.

#Main configuration file – amanda.conf
Most configuration information goes into the amanda.conf configuration file. The configuration files are kept into a place /etc/amanda/*configname* which for me is /etc/amanda/normal.

org "Example Company"   # Title of report
mailto "root"           # recipients of report, space separated
dumpuser "backup"       # the user to run dumps under
inparallel 4            # maximum dumpers that will run in parallel
netusage  600           # maximum net bandwidth for Amanda, in KB per sec

# a filesystem is due for a full backup once every  days
dumpcycle 4 weeks       # the number of days in the normal dump cycle
tapecycle 8 tapes       # the number of tapes in rotation

bumpsize 20 MB          # minimum savings (threshold) to bump level 1 > 2
bumpdays     1          # minimum days at each level
bumpmult     4          # threshold = bumpsize * (level-1)**bumpmult

runtapes     1
tapedev "/dev/nst0"     # Linux @ tuck, important: norewinding

tapetype SDT-5000               # what kind of tape it is (see tapetypes below)
labelstr "^MY-TAPE-[0-9][0-9]*$"        # label constraint regex: all tapes must match

diskdir "/var/tmp"              # where the holding disk is
disksize 1000 MB                        # how much space can we use on it
infofile "/var/lib/amanda/normal/curinfo"       # database filename
logfile  "/var/log/amanda/normal/log"   # log filename

# where the index files live
indexdir "/var/lib/amanda/normal/index"

define tapetype SDT-5000 {
    comment "Sony SDT-5000"
    length 1584 mbytes
    filemark 0 kbytes
    speed 271 kps
}
define dumptype comp-home-tar {
    program "GNUTAR"
    comment "home partition dump with tar"
    options compress-fast, index, exclude-list "/etc/amanda/normal/home.exclude"
   priority medium
}

define dumptype comp-var-tar {
    program "GNUTAR"
    comment "var partition dump with tar"
    options compress-fast, index, exclude-list "/etc/amanda/normal/var.exclude"
    priority high
}

So what do all those lines mean? You can leave most of them what they are
in this example. Read the amanda(8) manual page for information about
what each of the lines does. I’ll only point out some of the more significant
or tricky ones here.

runtapes
I have runtapes set to 1 because it is a single tape
drive and not some fancy multi-drive tape jukebox.
tapedev
This is the Unix device that the tape is found. Try
/dev/nst0 first as its a good default. Use dmesg to find the device if
that doesn’t work.
tapetype
This refers to a label that is defined further along
in the configuration file.
labelstr
All tape labels have to match this regular expression. AMA
NDA wont recognize it otherwise.
diskdir
A directory where AMANDA can temporarily store its files
before they are put onto the tape, not sure if /var/tmp is a good idea or not.
disksize
The amount of space that AMANDA can use in the previously
given diskdir.
infofile, logfile, indexdir
Filenames for storing information and
status of your backups. All these directories for these files need to exist,
see below.

The tapetype definition have been previously explained. It’s either going to be a cut and paste from a website or the output of amtapetype.

The dumptypes depart from the usual ones you see in the amanda documentation. I am using tar here because then I can select what files and directories I want in the archive. The two dumptypes are identical except for the priority and the exclude-list.

#disklist config file
The disklist file is found in */etc/amanda/normal/* directory and it lists what disks on what hosts are to be backed up using what backup format. Each line has three entries separated by whitespace: hostname, drive or partition and dumptype. The dumptype is one of the ones that was defined in the configuration file amanda.conf. My disklist looks like:

# disklist for normal backups
# Located at /etc/amanda/normal/disklist
#
localhost /var  comp-var-tar
localhost /home comp-home-tar

Reading both this file and the amanda.conf file, you can see that this means /var is backed up using tar and gzip and it will use the exclusion file var.exclude and /home is backed up also using tar and gzip but with the
exclusion file home.exclude.

#The tar exclude files
I create two exclude files, one for each partition type I am backing up. The contents of these exclude files are a file or directory name, one per line. You may want to read the GNU tar info files about what can go here, its just using the –exclude-from flag. My var.exclude excludes logs, debian packages, cached processed manual pages and temporary files, it looks like this:

/logs/*/*.gz
/cache/apt/archives/*.deb
cache/man/man
tmp

#Creating directories and files
There are a fair few directories and files that AMANDA needs to have setup with the correct permissions before it will work. The easiest way to document this is to show you the commands used. Remember that my config name is “normal” and a lot of these directories are defined in the amanda.conf file as is the user.

server# chown backup.backup /etc/amanda/normal
server# chmod 770 /etc/amanda/normal
server# touch /etc/amanda/normal/tapelist
server# chown backup.backup /etc/amanda/normal/*
server# chmod 500 /etc/amanda/normal/*
server# touch /var/lib/amanda/amandates
server# chown backup.backup /var/lib/amanda/amandates
server# mkdir /var/lib/amanda/normal
server# mkdir /var/lib/amanda/normal/index
server# chown -R backup.backup /var/lib/amanda/normal
server# chmod -R 770 /var/lib/amanda/normal
server# mkdir /var/log/amanda/normal
server# chown backup.backup /var/log/amanda/normal
server# chmod 770 /var/log/amanda/normal

#Making a new AMANDA tape
You will now need to make a new tape and label it. Put the tape in the drive and use the command

> amlabel normal MY-TAPE-01

Remember the label has to match the regular expression you have in the amanda.conf file. If all goes well, in a minute or two you will have a tape ready.

#Checking the configuration
AMANDA has a checking program, called amcheck, that makes sure everything is ready to go for the backup. The example output given below shows most things are ok, but I need to change a tape (and run amlabel on it) because I’ve already used this tape.

# su backup
$ amcheck normal
Amanda Tape Server Host Check
-----------------------------
Holding disk /var/tmp: 6349516 KB disk space available, that's plenty
ERROR: cannot overwrite active tape MY-TAPE-01
       (expecting a new tape)
NOTE: skipping tape-writable test
Server check took 9.027 seconds

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.032 seconds, 0 problems found

(brought to you by Amanda 2.4.4)

#To do the backup
After all that setup and testing, the backup itself is, well, pretty boring. To start it, as the backup user type /usr/sbin/amdump normal. You probably want this in a crontab so it is done regularly. After some time whoever was mentioned in the mailto line in the amanda.conf file will get an email about the backup.

#Recovering a file
To test the backup, you really need to test that you can restore a file. To do this I cd to /tmp and then run the amrecover utility. The amrecover program looks a lot like a ftp client and is pretty easy to use. Once again, it needs the config name so to run it, type amrecover normal. It may not know what host you want to go to, so set the host with the sethost command.

amrecover> sethost localhost
200 Dump host set to localhost.

Next you need to tell the recover program what disk you are trying to recover. We’ll select the /home directory here.

amrecover>; setdisk /home
Scanning /var/tmp...
20030618: found Amanda directory.
200 Disk set to /home.

We’ve found the disk and the host, now to wander around the filesystem and find the file. The file we are after is /home/csmall/myfile.txt. We’ve already at /home with the setdisk command, so now it is just a matter of cd into the csmall directory, add the file myfile.txt to the recovery list and then issue the extract command to start the recovery process.

amrecover> cd csmall
/home/csmall
amrecover> add myfile.txt
Added /csmall/myfile.txt
amrecover> extract

Extracting files using tape drive /dev/nst0 on host localhost.
The following tapes are needed: MY-TAPE-01

Restoring files into directory /var/tmp
Continue [?/Y/n]?

The restore program has found the file you are after and what tapes are needed to restore the file. If there were many files there may be multiple tapes required. It will now ask you to continue and after pressing
enter it will pause while you load the required tape into the drive. Pressing enter again will restore the file. It may take an hour or so to get it.

Extracting files using tape drive /dev/nst0 on host localhost.
Load tape MY-TAPE-01 now
Continue [?/Y/n/s/t]?
./csmall/myfile.txt

amrecover>

The result should be now a restored csmall/myfile.txt in your current directory. If that’s the case, the restore test has succeeded.

Creating an APT archive

[apt](http://packages.debian.org/apt) is a very important and useful tool that is used mainly for [Debian](http://www.debian.org/) GNU/Linux computers to download and install packages. It has the ability of sorting out the dependencies for packages and downloading from multiple sites.

For various reasons, people want to run their own apt archive that is separate from the rest of the Debian package distribution system. This gives a much better way of distributing binary and source packages that just a plain FTP or HTTP site.

For this document, I have used my archive hosted on Internode as the example. Apparently there is a way of doing this using the new pool method. I couldn’t get it to work and junked it and went back to the old way which was putting the packages under dist. It seems a lot cleaner and the reasons for having /pool/ don’t really apply for small archives.

#Definitions
To understand how apt (or Debian for that matter) sorts its files, you need to understand the various ways files are catalogued. This will help in deciding what to call the various directories.

Dist
– The distribution of Debian. Can either be a code-word like woody, sid or sarge or a type like stable, testing or unstable. For my archive I use unstable. Not that some places DIST is the directory dist/distname such as dist/unstable.
Section
– The section determines what is the state of the package and is determined by the copyright. DFSG-free packages go into the main section.
Arch
– What architecture is the package built for?

#Directory Layout
Apt requires a certain type of directory layout to work. The directories can either be real directories or symlinks. This is what my archive looks like:

apt/
  +-dists/
    +-unstable/
      +-main/
        +-binary-i386/
          +-Packages
          +-Packages.gz
          +-gkrellm-wmium_1.0.8-1_i386.deb
          +-wmium_1.0.8-1_i386.deb
        +-source/
          +-wmium_1.0.8-1.diff.gz
          +-wmium_1.0.8-1.dsc
          +-wmium_1.0.8.orig.tar.gz

The binaries are found in the sub-directory *./apt/dists/unstable/main/binary-i386/* while the source packages are found in *./apt/dists/unstable/main/source/*

#apt-ftparchive configuration file
The most difficult part of the whole exercise is trying to get this configuration file right. It’s badly documented and has no real examples combined with the fact if something doesn’t work you don’t know why.
I call mine archive.conf but it doesn’t really matter what it is called as long as you use the same name when you run the programs in the next steps. After much trial and error, I have the following configuration file, explanations of what the lines do follows.

Dir {
  ArchiveDir "/home/example/myarchive/apt";
};

BinDirectory "dists/unstable" {
  Packages "dists/unstable/main/binary-i386/Packages";
  SrcPackages "dists/unstable/main/source/Sources";
};
ArchiveDir
The absolute path to the top of the archive from the server’s point of view. This directory will have the dists directory in it. Now if you are building the files on one machine but uploading them to another (like I do) then the directory is the directory for the building machine.
BinDirectory
This is the directory of the dist, that directory only has the main symlink in it.
Packages
The location of the Packages file. The full path will be

SrcPackages
The location of the Source Packages file. The full path
will be $ArchiveDir/$BinDirectory/main/$Packages

#Adding Packages
To add packages, put the .deb files into the binary-i386 directory and the orig.tar.gz, .dsc and diff.gz files into the source directory.

#Running apt-ftparchive
To update or create the Packages filenames, you need to run apt-ftparchive. The programs scans for packages and creates the right paths in the Packages file for them.

$ apt-ftparchive generate archive.conf
 dists/unstable: 2 files 3017kB 0s
Done Packages, Starting contents.
Done. 3017kB in 2 archives. Took 0s

Notice it has found 2 files and 2 archives, which means it is working because that was the number of packages I had in my archive. You should also have a Packages and Packages.gz in the binary-i386 directory.

#Uploading the archive
If you are using the same computer for creating the archive then you are done. If not then then you need to move the files onto the server. How you do this depends on what the server has available. Ideally, they
have scp or rsync which makes it very easy. My ISP only has FTP which means I need something like [lftp](href=”http://packages.debian.org/lftp) to do the copying.

 $ lftp -c 'open -u myusername ftp.myisp.net ; mirror -n -R apt apt'

This command recursively copies files from the local apt directory to the remote apt directory on the ftp server. See the lftp manual page for details.

#sources.list changes
Now you have a working archive, you need to change your /etc/apt/sources.list file so that apt knows to get packages from your archive. It looks like just another archive.

deb http://users.on.net/csmall/apt unstable main

#My Makefile
The following is my Makefile that sits at the top directory (the same directory that the apt subdirectory sits in on the local computer) that I use to make the various files.

instpkg:
  -mv incoming/*_i386.deb apt/dists/unstable/main/binary-i386/
  -mv incoming/*.dsc incoming/*.diff.gz incoming/*.orig.tar.gz apt/dists/unstable/main/source/
  apt-ftparchive generate archive.conf

lftp:
  lftp -c 'open -u myself ftp.isp.net ; mirror -n -R apt apt'