Tag: backups

  • Using Amanda for Backups

    [AMANDA](http://www.amanda.org/) as the name says, is an advanced disk archiver or, more simply, a backup program. It’s a really useful program, let down by something approaching the worst documentation out there. I’ve had about 4 attempts on and off for about the same number of years at using it and deciphering the documentation. Finally I got it to work, but it should not be that hard. Who knows, maybe in
    another 4 years I won’t hate Wiki documentation? Naah.

    This document shows you what files you need to create, their format and how to use some of the programs. It’s probably only a brief get you started type of document, but at least you should be able to start. I have tested this setup on a bunch of [Debian](http://www.debian.org/) GNU/Linux servers but it should be reasonably similar for other systems.

    Speaking of my systems, what I’m writing here works for me with my setup. It’s quite likely it will work for you, but you should verify and test everything. If something is not quite right and you lose all your valuable
    files because something is wrong in here, you should of tested it properly yourself. In any case, a non-tested backup system is almost as useless as a non-working one.

    #Determine your backup regime and other details
    The first step is to decide what you want to backup. Do you want to do everything? Just home files? The variety is quite large and it is just a matter of working out how to tell AMANDA how to do it. AMANDA works
    in chunks of partitions, so if you have everything in one partition as / then that’s one big chunk. All is not lost however as you can use some tricks to exclude some type of files or directories.

    For my setup, I backup all of /, most of /var except logs and cache and /home with the exception of some junk. The method for backing up all of a partition can be done differently than backing up part of it. I’m not
    sure, but I think its quicker to backup all of a partition when you can.

    Next, think of a name to call the backup regime. I call mine “normal” and thats what I’ll use here. AMANDA calls the name “the config” in the manual pages. It’s just a label for the set of configuration files.

    #Determining tapetype

    The configuration file (see next section) will need to know the tapetype. If you don’t know what drive model you have, you can often get some information about the device in the **/proc/scsi/scsi** file. My file shows my ancient Sony tape drive.

    Attached devices:
    Host: scsi0 Channel: 00 Id: 01 Lun: 00
      Vendor: SONY     Model: SDT-5000&
    nbsp;        Rev: 3.30
      Type:   Sequential-Access    &nbs
    p;           ANSI SC
    SI revision: 02

    The tapetype is a bunch of parameters about your tape drive and they are different for
    every device. Have a look at [AMANDA tapetype list](http://amanda.sourceforge.net/fom-serve/cache/45.html) to see if yours is there.

    Failing that, use the **amtapetype** program. Be warned, it will take a long time for this program to work. When I mean a long time, I’m talking about 2 hours or so. That’s roughly how long it took for me.

    To run it, type:

    server# amtapetype /dev/nst0
    

    and then wait, and wait… Eventually you’ll get something that you can cut and paste into your amanda.conf /dev/st0 is the device the tapedrive uses, try that one first if you have a SCSI tape drive.

    #Main configuration file – amanda.conf
    Most configuration information goes into the amanda.conf configuration file. The configuration files are kept into a place /etc/amanda/*configname* which for me is /etc/amanda/normal.

    org "Example Company"   # Title of report
    mailto "root"           # recipients of report, space separated
    dumpuser "backup"       # the user to run dumps under
    inparallel 4            # maximum dumpers that will run in parallel
    netusage  600           # maximum net bandwidth for Amanda, in KB per sec
    
    # a filesystem is due for a full backup once every  days
    dumpcycle 4 weeks       # the number of days in the normal dump cycle
    tapecycle 8 tapes       # the number of tapes in rotation
    
    bumpsize 20 MB          # minimum savings (threshold) to bump level 1 > 2
    bumpdays     1          # minimum days at each level
    bumpmult     4          # threshold = bumpsize * (level-1)**bumpmult
    
    runtapes     1
    tapedev "/dev/nst0"     # Linux @ tuck, important: norewinding
    
    tapetype SDT-5000               # what kind of tape it is (see tapetypes below)
    labelstr "^MY-TAPE-[0-9][0-9]*$"        # label constraint regex: all tapes must match
    
    diskdir "/var/tmp"              # where the holding disk is
    disksize 1000 MB                        # how much space can we use on it
    infofile "/var/lib/amanda/normal/curinfo"       # database filename
    logfile  "/var/log/amanda/normal/log"   # log filename
    
    # where the index files live
    indexdir "/var/lib/amanda/normal/index"
    
    define tapetype SDT-5000 {
        comment "Sony SDT-5000"
        length 1584 mbytes
        filemark 0 kbytes
        speed 271 kps
    }
    define dumptype comp-home-tar {
        program "GNUTAR"
        comment "home partition dump with tar"
        options compress-fast, index, exclude-list "/etc/amanda/normal/home.exclude"
       priority medium
    }
    
    define dumptype comp-var-tar {
        program "GNUTAR"
        comment "var partition dump with tar"
        options compress-fast, index, exclude-list "/etc/amanda/normal/var.exclude"
        priority high
    }
    

    So what do all those lines mean? You can leave most of them what they are
    in this example. Read the amanda(8) manual page for information about
    what each of the lines does. I’ll only point out some of the more significant
    or tricky ones here.

    runtapes
    I have runtapes set to 1 because it is a single tape
    drive and not some fancy multi-drive tape jukebox.
    tapedev
    This is the Unix device that the tape is found. Try
    /dev/nst0 first as its a good default. Use dmesg to find the device if
    that doesn’t work.
    tapetype
    This refers to a label that is defined further along
    in the configuration file.
    labelstr
    All tape labels have to match this regular expression. AMA
    NDA wont recognize it otherwise.
    diskdir
    A directory where AMANDA can temporarily store its files
    before they are put onto the tape, not sure if /var/tmp is a good idea or not.
    disksize
    The amount of space that AMANDA can use in the previously
    given diskdir.
    infofile, logfile, indexdir
    Filenames for storing information and
    status of your backups. All these directories for these files need to exist,
    see below.

    The tapetype definition have been previously explained. It’s either going to be a cut and paste from a website or the output of amtapetype.

    The dumptypes depart from the usual ones you see in the amanda documentation. I am using tar here because then I can select what files and directories I want in the archive. The two dumptypes are identical except for the priority and the exclude-list.

    #disklist config file
    The disklist file is found in */etc/amanda/normal/* directory and it lists what disks on what hosts are to be backed up using what backup format. Each line has three entries separated by whitespace: hostname, drive or partition and dumptype. The dumptype is one of the ones that was defined in the configuration file amanda.conf. My disklist looks like:

    # disklist for normal backups
    # Located at /etc/amanda/normal/disklist
    #
    localhost /var  comp-var-tar
    localhost /home comp-home-tar
    

    Reading both this file and the amanda.conf file, you can see that this means /var is backed up using tar and gzip and it will use the exclusion file var.exclude and /home is backed up also using tar and gzip but with the
    exclusion file home.exclude.

    #The tar exclude files
    I create two exclude files, one for each partition type I am backing up. The contents of these exclude files are a file or directory name, one per line. You may want to read the GNU tar info files about what can go here, its just using the –exclude-from flag. My var.exclude excludes logs, debian packages, cached processed manual pages and temporary files, it looks like this:

    /logs/*/*.gz
    /cache/apt/archives/*.deb
    cache/man/man
    tmp
    

    #Creating directories and files
    There are a fair few directories and files that AMANDA needs to have setup with the correct permissions before it will work. The easiest way to document this is to show you the commands used. Remember that my config name is “normal” and a lot of these directories are defined in the amanda.conf file as is the user.

    server# chown backup.backup /etc/amanda/normal
    server# chmod 770 /etc/amanda/normal
    server# touch /etc/amanda/normal/tapelist
    server# chown backup.backup /etc/amanda/normal/*
    server# chmod 500 /etc/amanda/normal/*
    server# touch /var/lib/amanda/amandates
    server# chown backup.backup /var/lib/amanda/amandates
    server# mkdir /var/lib/amanda/normal
    server# mkdir /var/lib/amanda/normal/index
    server# chown -R backup.backup /var/lib/amanda/normal
    server# chmod -R 770 /var/lib/amanda/normal
    server# mkdir /var/log/amanda/normal
    server# chown backup.backup /var/log/amanda/normal
    server# chmod 770 /var/log/amanda/normal
    

    #Making a new AMANDA tape
    You will now need to make a new tape and label it. Put the tape in the drive and use the command

    > amlabel normal MY-TAPE-01

    Remember the label has to match the regular expression you have in the amanda.conf file. If all goes well, in a minute or two you will have a tape ready.

    #Checking the configuration
    AMANDA has a checking program, called amcheck, that makes sure everything is ready to go for the backup. The example output given below shows most things are ok, but I need to change a tape (and run amlabel on it) because I’ve already used this tape.

    # su backup
    $ amcheck normal
    Amanda Tape Server Host Check
    -----------------------------
    Holding disk /var/tmp: 6349516 KB disk space available, that's plenty
    ERROR: cannot overwrite active tape MY-TAPE-01
           (expecting a new tape)
    NOTE: skipping tape-writable test
    Server check took 9.027 seconds
    
    Amanda Backup Client Hosts Check
    --------------------------------
    Client check: 1 host checked in 0.032 seconds, 0 problems found
    
    (brought to you by Amanda 2.4.4)
    

    #To do the backup
    After all that setup and testing, the backup itself is, well, pretty boring. To start it, as the backup user type /usr/sbin/amdump normal. You probably want this in a crontab so it is done regularly. After some time whoever was mentioned in the mailto line in the amanda.conf file will get an email about the backup.

    #Recovering a file
    To test the backup, you really need to test that you can restore a file. To do this I cd to /tmp and then run the amrecover utility. The amrecover program looks a lot like a ftp client and is pretty easy to use. Once again, it needs the config name so to run it, type amrecover normal. It may not know what host you want to go to, so set the host with the sethost command.

    amrecover> sethost localhost
    200 Dump host set to localhost.
    

    Next you need to tell the recover program what disk you are trying to recover. We’ll select the /home directory here.

    amrecover>; setdisk /home
    Scanning /var/tmp...
    20030618: found Amanda directory.
    200 Disk set to /home.
    

    We’ve found the disk and the host, now to wander around the filesystem and find the file. The file we are after is /home/csmall/myfile.txt. We’ve already at /home with the setdisk command, so now it is just a matter of cd into the csmall directory, add the file myfile.txt to the recovery list and then issue the extract command to start the recovery process.

    amrecover> cd csmall
    /home/csmall
    amrecover> add myfile.txt
    Added /csmall/myfile.txt
    amrecover> extract
    
    Extracting files using tape drive /dev/nst0 on host localhost.
    The following tapes are needed: MY-TAPE-01
    
    Restoring files into directory /var/tmp
    Continue [?/Y/n]?
    

    The restore program has found the file you are after and what tapes are needed to restore the file. If there were many files there may be multiple tapes required. It will now ask you to continue and after pressing
    enter it will pause while you load the required tape into the drive. Pressing enter again will restore the file. It may take an hour or so to get it.

    Extracting files using tape drive /dev/nst0 on host localhost.
    Load tape MY-TAPE-01 now
    Continue [?/Y/n/s/t]?
    ./csmall/myfile.txt
    
    amrecover>
    

    The result should be now a restored csmall/myfile.txt in your current directory. If that’s the case, the restore test has succeeded.