1 Introduction

The tob script and its setup is based on the Icmake-driven backup scripts, also by me. I rewrote the scripts to a bash- interpreted form mainly to gain peformance (Icmake interpretation was, alas, significantly slower).

If you are a user of the no longer supported backup scripts, you might want to switch to tob: the installation is easier, and a better configurability is possible. The volume definition files which the backup scripts introduced can (with slight modifications) be used for tob.

History:

• 0.00 First version for testing purposes.
• 0.01 First released version, including resources for afio and tar-based backups.
• 0.02 Resource file for remote afio backups added.
• 0.03 The POSTCMD is run only once, following a backup command or due to the cleanup function.
• 0.04 FILESPEC implemented in the restore option. Buggy restore fixed.
• 0.05 and 0.06 Quoted many shell variables to avoid premature expansion. Filenames with spaces should be OK now.
• 0.07 Incremental backup facility added. TYPE variable for resource files defined.
• 0.08 Filenames with backslashes should be properly recognized.
• 0.09 to 0.11 Unreleased maintenance versions, support for files with spaces included.
• 0.12 Incremental backups implemented.
• 0.13 Fixed directory permissions in afio-based backups. The directories themselves are stored too.

Before describing the installation and usage of tob, a few general notes about backups are in order.

The backup program

A backup is always created (or restored) using a backup program, also called the archiver. The choice of the backup program can be quite important: some programs produce more error-prone backups than others.

Typical backup programs for Unix environments are cpio, tar and the newer afio. cpio is a program which accepts a list of files on stdin and copies these files to a backup device. The list of files is typically constructed with the program find. A disadvantage of cpio is that the files are not written in compressed format.

tar, the `Tape ARchiver', generates a list of files itself and writes the files to a backup device. Newer versions of tar support the -z flag, which specifies that the data stream is filtered through a compression program (typically gzip) before it is sent to the backup device. An advantage of this method is, obviously, that the backup is smaller. An important disadvantage is the fact that the entire backup is compressed, from the first byte to the last. If the backup medium develops a fault (e.g., a bad sector on a diskette or a tape) then the entire backup beyond the fault becomes unreadable: as this section cannot be uncompressed, the files in it cannot be identified.

The last backup program which is described here is afio. This program is in its usage similar to cpio, except that it supports a -z flag which instructs it to filter individual files through a compression program. The final backup then consists of afio's uncompressed headers and compressed data. Using this approach, afio presents `the best of two worlds': compressed backups which however are not too error-prone. When a fault develops on the backup medium, afio can skip the invalid data until it encounters a new header; files beyond the fault can still be accessed. afio 2.4.0 seemed to malfunction while restoring symbolic links (thanks, Berthold Crysmann (crysmann@rrz.uni-hamburg.de)), but 2.4.1 looks all right.

The tob setup includes a number of sample definition files, which select compressed tar backups, uncompressed tar backups or compressed afio backups. The tob package somewhat shows my preferences for afio, as this is the default setting.

Full, differential and incremental backups

Backup types are often described as full, differential or incremental backups. A full backup consists of all files of a given set.

A differential backup consists of all files of a set which were changed since the last full backup. The tob package selects files for a differential backup using a ``hungry'' approach: files are considered `changed' not only by date comparison, but also by comparing size information, inode information etc.. This means that when two files are swapped, as in

 # swap two files /etc/a and /etc/b
 mv /etc/a /tmp/x
 mv /etc/b /etc/a
 mv /tmp/x /etc/b

tob will (correctly) flag these files as `changed' and select them for a differential backup. Compare this to an approach in which `differential' backups consist only of files that are newer than a certain date: those backups would not include two swapped files.

This setup of different types of backups means that tob has to maintain lists of backed up files. These lists are kept in a compressed format. Tob keeps one file of each type (one listing of the last full backup, one listing of the last differential backup and listings of incremental backups). This approach has the following philosophy in mind: when a full backup is created, all files are saved on a backup medium. A listing of the previous full backup therefore becomes obsolete; why restore files from a stale full backup when a more recent version exists? The same reasoning holds true for differential backups; only the most recent one should be used when restoring files. Therefore, by definition, only one version of the last full or differential backup may exist since this version is the most recent. Tob automatically keeps only the last listings of a full or differential backup, all other listings are automatically deleted. Once a full and a differential backup is present, all files can be restored from these two backups.

Incremental backups are a tad more difficult. Incremental backups are defined as consisting of all files which are changed since all previous backups: be it full, differential or incremental. Typically, incremental backups are made more frequently than the other types and are stored each on their own medium. Tob maintains a listing for each incremental backup ever made, until a full or differential backup is scheduled. A possible backup scheme is the following:

• To begin with, a full backup of a filesystem should be created. This backup is likely to be large. For simplicity's sake, let's assume that you have five files a, b, c, d and e on your hard disk: the full backup will contain all files a to e.

  The most recent state of files is now of course kept on the medium which contains the full backup. Tob saves a listing of the backed up files.

• At regular intervals, say once a day or once a week, an incremental backup should be created. By definition, an incremental backup will store all files which are changed since all previous backups: at this stage, an incremental backup will store all files that are changed since the full backup and since all other incremental backups. E.g., say that you change file b and make your first incremental backup. Then this incremental backup will hold only file b. Next, you change c and make another incremental backup. That one will hold only file c. By now, you should have three backup media (be it tapes, diskettes or whatever): one with a full backup of files a to e, one with an incremental backup of b and one with an incremental backup of c.

  The most recent state of files is therefore on the full backup and on all subsequent incremental backups. Tob saves listings of the backed up files in listings: one for the full backup and one for each incremental backup. This approach also means that the restoring of a file or of a set of files means that the operator must check the listing files to see which backup contains the most recent version of the file. E.g., a file may exist on the full backup and on two incremental backups -- in that case, the file was first backed up on the full backup and then changed twice.

• At a given time too many incremental backups will exist. E.g., when the backups are written to a tape, all tapes may be full. At this moment a differential backup should be scheduled: this will contain all files since the last full backup. In the above example, a differential backup will now hold files b and c. The two incremental backups hence become obsolete -- thereby freeing your backup media for next usage.

  The most recent state of files is now kept on the full backup and on the differential backup. The series of the incremental backups therefore becomes outdated: tob will now only save the listings of the full and differential backup and automatically remove the listings of the incremental backups.

• The series of the incremental backups can now be re-enabled. As files are changed, they are again written to the incremental backups; until again a differential backup should be made. E.g., you could modify file d and run an incremental backup. That backup would hold only file d; the full set of files could be restored using the full backup (for files a and e), the differential backup (for b and c) and the one incremental backup (for file d).
• Generally the size of a differential backup should determine whether a new full backup should be made. E.g., a criterium might be that the differental backup no longer fits on one tape. When a new full backup is created, all previous backups become outdated and the whole procedure is started from the top.

The choice of the `ideal' backup schedule of course depends on the setup. I for myself just make full backups, because all files on my Linux partition fit on one tape. I hardly ever make differentials; let alone incrementals.

Please don't mail me with questions or bug reports about tob unless you have checked on ftp.icce.rug.nl. It is very well possible that a `bug-repaired' archive of tob is present on the primary site. If you do find a bug in the latest version, then of course please do mail me.

The tob archive may furthermore be found at the `standard' Linux sites sunsite.unc.edu and tsx-11.mit.edu. These sites also carry the backup program afio.

The current tob setup knows only one bug (thanks, Warwick Allison (warwick@cs.uq.oz.au) and Peter Mutsaers (plm@atcmp.nl), also for other bug reports and countless improvements).

Since this behavior is now documented, it's no longer a bug but a quirk. Maybe even a feature. So there you are.

Another previous bug is now fixed (thanks, Peter): files with spaces in the names are now handled correctly. Furthermore, older versions of tob restored directories with wrong permissions, that's fixed too (thanks, Jeff Coy Jr.).