Backup with Bacula

In this tutorial, I will explain and show how to install and configure bacula-5.0.3 backup software based on the following setup:

Environment:

1) SuSE Linux Enterprise Server 11 for IBM PPC (LPAR)

    You may find many or most of documentations on bacula are for intel x86/64.

2) Bacula version bacula-5.0.3 as per this writing, this is not the enterprise version with  support available.

3) IBM TS3200 tape library with two drives inside and 45 slots

4) 2 LT04 tapes, 500GB each

Prior to installing bacula, I have to check and make sure the Operating system and backup devices are in place and work accordingly. I will not explain how to connect the tape library to the server. Once the library is on, slot in the tapes inside the slots and we are ready.

From here, anything including moving the tapes into different slot, erase etc are done via the shell (bash).

Installation:

Bacula Requirements:

Install the following packages from the SLES installer (DVD)

- c & c++ compiler

- mysql server (mysql)

- mysql development (mysql-devel)

- QT (graphical components)

1) Download the tarball package from http://www.bacula.org/en/?page=downloads

2) extract the package, tar zxvf bacula-5.0.3.tar.gz

3) cd to the bacula-5.0.3 directory, cd bacula-5.0.3

4) configure and install bacula

For latest version download go to Bacula latest

Compiling Bacula

./configure --sbindir=/usr/local/bacula/bin --sysconfdir=/usr/local/bacula/bin --with-pid-dir=/usr/local/bacula/bin/working --with-subsysdir=/usr/local/bacula/bin/working --enable-smartalloc --with-mysql --with-working-dir=/usr/local/bacula/ --with-dump-email=yourid@yourdomain.com --with-job-email=yourid@yourdomain.com --with-smtp-host=localhost --enable-bat --with-qwt=/usr/include/qwt --enable-bwx-console  --enable-tray-monitor

make
make install
make install-autostart

the 'make install-autostart' will by default set bacula to auto start at runlevel 3 & 5

The installation destination is 

/usr/local/bacula/

note: I have encountered an error message during bacula startup, this is because the program will write pid files in a directory call working inside the bin directory, for that purpose create the directory.

mkdir -p /usr/local/bacula/bin/working

Testing the hardware (tape library and tape drive)

1) install the mtx package from SuSE add remove program/yast or yast2

note: mtx package is required by bacula scripts, later on you may find the usage.

2) check the device name in the suse hardware information via yast or yast2

Below are the device listing of my setup:

tape library is /dev/sg6

tape drive no. 1 is /dev/st1

tape drive no. 2 is /dev/st2

3) List the drives, slot and tapes inside the library, the following shows 2 tapes inserted into slot number 44 and 45 respectively.

mtx -f /dev/sg6 status

  

Output from mtx command

mtx: Request Sense: Long Report=yes
mtx: Request Sense: Valid Residual=no
mtx: Request Sense: Error Code=70 (Current)
mtx: Request Sense: Sense Key=Unit Attention
mtx: Request Sense: FileMark=no
mtx: Request Sense: EOM=no
mtx: Request Sense: ILI=no
mtx: Request Sense: Additional Sense Code = 29
mtx: Request Sense: Additional Sense Qualifier = 00
mtx: Request Sense: BPV=no
mtx: Request Sense: Error in CDB=no
mtx: Request Sense: SKSV=no
Mode sense (0x1A) for Page 0x1D failed
Storage Changer /dev/sg6:2 Drives, 45 Slots ( 0 Import/Export )
Data Transfer Element 0:
Data Transfer Element 1:
Storage Element 1:Empty
Storage Element 2:Empty
Storage Element 3:Empty
Storage Element 4:Empty
Storage Element 5:Empty
Storage Element 6:Empty
Storage Element 7:Empty
Storage Element 8:Empty
Storage Element 9:Empty
Storage Element 10:Empty
Storage Element 11:Empty
Storage Element 12:Empty
Storage Element 13:Empty
Storage Element 14:Empty
Storage Element 15:Empty
Storage Element 16:Empty
Storage Element 17:Empty
Storage Element 18:Empty
Storage Element 19:Empty
Storage Element 20:Empty
Storage Element 21:Empty
Storage Element 22:Empty
Storage Element 23:Empty
Storage Element 24:Empty
Storage Element 25:Empty
Storage Element 26:Empty
Storage Element 27:Empty
Storage Element 28:Empty
Storage Element 29:Empty
Storage Element 30:Empty
Storage Element 31:Empty
Storage Element 32:Empty
Storage Element 33:Empty
Storage Element 34:Empty
Storage Element 35:Empty
Storage Element 36:Empty
Storage Element 37:Empty
Storage Element 38:Empty
Storage Element 39:Empty
Storage Element 40:Empty
Storage Element 41:Empty
Storage Element 42:Empty
Storage Element 43:Empty
Storage Element 44: 910ACYL4
Storage Element 45: 911ACYL4

Bacula Components

1.Director daemon

This daemon co-ordinate all working of backup,and through its configuration file we can specify all these things.

2.File daemon

This daemon works in all clients from that client we are backup data. Director daemon connect to this daemon after authentication and backup the files from this client.

3.Storage daemon

This daemon is for store the backup data from client in to hard disk of backup server,usually this daemon and director daemon works in the same backup server. director works as intermediate between the file daemon and storage daemon.

4.Console daemon

This is a terminal to control all works.This console connect to director daemon and using its commands we can define all things related with backup .

5.Catalog Database

The database used here is for store all information related to the backup, including the file indexing.Commonly used database for bacula is Mysql.
This figure shows how the different bacula daemon configuration files were linked together.

Source: http://www.bacula.org/5.2.x-manuals/en/main/main/img7.png

Modify bacula config files

Before we can start the bacula service and use the scripts in the bin directory, we must first configure the configuration files. There are three files involved:

1. bacula-sd.conf (storage daemon, only configured on the bacula server)

2. bacula-dir.conf (director daemon, only configured on the bacula server)

3. bacula-console.conf (console configuration file, you will use console to manage backup activities, alternatively use BAT Gui)

4. bacula-fd.conf (file daemon, configured on the host you want to backup)

The bacula-sd.conf is where you configure the backup devices to be used, this include the library and drives.

Below are the configuration I have made to my bacula-sd.conf

#start

Storage {                             

  Name = p770vlpar3-sd    # name of this storage daemon

  SDPort = 9103                  # Director's port      

  WorkingDirectory = "/usr/local/bacula/"

  Pid Directory = "/usr/local/bacula/bin/working"

  Maximum Concurrent Jobs = 20

}

# List Directors who are permitted to contact Storage daemon

Director {

  Name = p770vlpar3-dir

  Password = "oscc123"

}

# Restricted Director, used by tray-monitor to get the

#   status of the storage daemon

Director {

  Name = p770vlpar3-mon

  Password = "oscc123"

  Monitor = yes

}

# An autochanger device with two drives 

Autochanger {

  Name = ibmts3200

  Device = Drive1,Drive2

  Changer Device = /dev/sg6

  Changer Command = "/usr/local/bacula/bin/mtx-changer %c %o %S %a %d"

}

Device {

  Name = Drive1                      

  Drive Index = 0

  Media Type = LTO-2

  Archive Device = /dev/st1

  AutomaticMount = yes;               

  AlwaysOpen = yes;

  RemovableMedia = yes;

  RandomAccess = yes;

  AutoChanger = yes;

  RequiresMount = yes

  #Maximum Volume Size=6G #uncomment this if you want to set limit of 6GB for a tape/volume.

  }

Device {

  Name = Drive2

  Drive Index = 1

  Media Type = LTO-2

  Archive Device = /dev/st2

  AutomaticMount = yes;              

  AlwaysOpen = yes;

  RemovableMedia = yes;

  RandomAccess = yes;

  AutoChanger = yes;

  RequiresMount = Yes

  #Maximum Volume Size=6G

  }

# Send all messages to the Director, 

# mount messages also are sent to the email address

Messages {

  Name = Standard

  director = p770vlpar3-dir = all

}

#End

Once done with modification of the bacula-sd.conf, save it and exit from editor.

Start bacula.

This will start all three daemons ( bacula-dir,bacula-sd and bacula-fd)

cd /usr/local/bacula/bin
./bacula start

Create mysql database for bacula

1) start mysql if not started, service mysql start

2) in the bacula bin directory, grant privilege to root user,

./grant_mysql_privileges -u root -p

3) create the database

./create_mysql_database -u root -p
./make_mysql_tables -u root -p

Now lets play with the tape library

Count how many slots in library without detail

cd /usr/local/bacula/bin
./mtx-changer /dev/sg6 slots


Load and unload tape to and from the drives:

Load tape from slot 44 to tape drive no. 1

./mtx-changer /dev/sg6 load 44 /dev/st0 0

Load tape from slot 45 to tape drive no. 2

./mtx-changer /dev/sg6 load 45 /dev/st1 1

Now, check the location of the tapes inside the library

mtx -f /dev/sg6 status

mtx: Request Sense: Long Report=yes
mtx: Request Sense: Valid Residual=no
mtx: Request Sense: Error Code=70 (Current)
mtx: Request Sense: Sense Key=Unit Attention
mtx: Request Sense: FileMark=no
mtx: Request Sense: EOM=no
mtx: Request Sense: ILI=no
mtx: Request Sense: Additional Sense Code = 29
mtx: Request Sense: Additional Sense Qualifier = 00
mtx: Request Sense: BPV=no
mtx: Request Sense: Error in CDB=no
mtx: Request Sense: SKSV=no
Mode sense (0x1A) for Page 0x1D failed
  Storage Changer /dev/sg6:2 Drives, 45 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = 911ACYL4                       
Data Transfer Element 1:Full (Storage Element 1 Loaded):VolumeTag = 910ACYL4
      Storage Element 1:Empty
      Storage Element 2:Empty
      Storage Element 3:Empty
      Storage Element 4:Empty
      Storage Element 5:Empty
      Storage Element 6:Empty
      Storage Element 7:Empty
      Storage Element 8:Empty
      Storage Element 9:Empty
      Storage Element 10:Empty
      Storage Element 11:Empty
      Storage Element 12:Empty
      Storage Element 13:Empty
      Storage Element 14:Empty
      Storage Element 15:Empty
      Storage Element 16:Empty
      Storage Element 17:Empty
      Storage Element 18:Empty
      Storage Element 19:Empty
      Storage Element 20:Empty
      Storage Element 21:Empty
      Storage Element 22:Empty
      Storage Element 23:Empty
      Storage Element 24:Empty
      Storage Element 25:Empty
      Storage Element 26:Empty
      Storage Element 27:Empty
      Storage Element 28:Empty
      Storage Element 29:Empty
      Storage Element 30:Empty
      Storage Element 31:Empty
      Storage Element 32:Empty
      Storage Element 33:Empty
      Storage Element 34:Empty
      Storage Element 35:Empty
      Storage Element 36:Empty
      Storage Element 37:Empty
      Storage Element 38:Empty
      Storage Element 39:Empty
      Storage Element 40:Empty
      Storage Element 41:Empty
      Storage Element 42:Empty
      Storage Element 43:Empty
      Storage Element 44:Empty
      Storage Element 45:Empty

Note: From the above listing there are two things you have to indentify

DTE

Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = 911ACYL4                       
Data Transfer Element 1:Full (Storage Element 1 Loaded):VolumeTag = 910ACYL4

1) Data Transfer Element 0  is the first drive, having volume/tape with barcode number 911ACYL4

2) Data Transfer Element 1  is the second drive, having volume/tape with barcode number 910ACYL4

SE

Storage Element 1:Empty
Storage Element 2:Empty
Storage element is the slot

Unload the tapes from the drive and return them to the slot

unload back tape from drive no.1 to slot no. 44

./mtx-changer /dev/sg6 unload 44 /dev/st0 0

unload back tape from drive no.2 to slot no. 45

./mtx-changer /dev/sg6 unload 45 /dev/st1 1

mtx -f /dev/sg6 status

Ok, that's how bacula manage the library.

Now lets move to bacula-dir.conf
I have made modification to the files with below configuration:

# Default Bacula Director Configuration file

Director {                           
  Name = p770vlpar3-dir     #p770vlpar3 is the server running bacula
  DIRport = 9101               
  QueryFile = "/usr/local/bacula/bin/query.sql"
  WorkingDirectory = "/usr/local/bacula/"
  PidDirectory = "/usr/local/bacula/bin/working"
  Maximum Concurrent Jobs = 10
  Password = "oscc123"        
  Messages = Daemon
}

# In this configuration, I have set few jobs including the bacula server itself, two other servers (system x and PSeries) are also defined below.
# Each backup job must have also a restore job.
# Pool is a group of tapes/volumes, grouped together with a pool name. In case you have large data to be backup, several tapes are needed. After first tape is full the second tape in the pool will be used automatically.

# START JOB FOR LOCALHOST

Job {                 #job for localhost
  Name = "p770vlpar3-fd-Job"
  Type = Backup
  Level = Full
  Client = p770vlpar3-fd     #localhost
  FileSet = "Oracle Full Set"
  Schedule = "WeeklyCycle"
  Storage = ibmts3200
  Messages = Standard
  Pool = oracle
  Priority = 1
  Write Bootstrap = "/usr/local/bacula/bin/working/p770vlpar3-fd.bsr"
}

Job {
  Name = "p770vlpar3-fd-Restore"
  Type = Restore
  Level = Full
  Client=p770vlpar3-fd                
  FileSet="Oracle Full Set"                 
  Storage = ibmts3200                   
  Pool = oracle
  Messages = Standard
  Where = /tmp/bacula-restores
  Bootstrap = "/usr/local/bacula/bin/working/p770vlpar3-fd.bsr"
}

# END JOB FOR LOCALHOST


# START JOB FOR SYSTEM X

Job {
  Name = "systemx2-Job"
  Type = Backup
  Level = Full
  Client = systemx2
  FileSet = "systemx2 Full Set"
  Schedule = "WeeklyCycle"
  Storage = ibmts3200
  Messages = Standard
  Pool = oracle
  Priority = 2
  Write Bootstrap = "/usr/local/bacula/bin/working/systemx2.bsr"
}

Job {
  Name = "systemx2-Restore"
  Type = Restore
  Level = Full
  Client= systemx2                
  FileSet="systemx2 Full Set"                 
  Storage = ibmts3200                   
  Pool = oracle
  Messages = Standard
  Where = /tmp/bacula-restores
  Bootstrap = "/usr/local/bacula/bin/working/systemx2.bsr"
}

# END JOB FOR SYSTEM X


# START JOB FOR PSERIES


Job {
  Name = "pseries-Job"
  Type = Backup
  Level = Full
  Client = pseries
  FileSet = "pseries Full Set"
  Schedule = "WeeklyCycle"
  Storage = ibmts3200
  Messages = Standard
  Pool = oracle
  Priority = 2
  Write Bootstrap = "/usr/local/bacula/bin/working/pseries.bsr"
}

Job {
  Name = "pseries-Restore"
  Type = Restore
  Level = Full
  Client= pseries                
  FileSet="pseries Full Set"                 
  Storage = ibmts3200                   
  Pool = oracle
  Messages = Standard
  Where = /tmp/bacula-restores
  Bootstrap = "/usr/local/bacula/bin/working/pseries.bsr"
}


# END JOB FOR PSERIES


# this is the target folder to be backup with compression level set to GZIP
# List of files to be backed up on localhost
FileSet {
  Name = "Oracle Full Set"
  Include {
    Options {
      signature = MD5
      compression = GZIP
            }
  
     File = /tmp   
        }

    }


# List of files to be backed up for systemx2
# For this client, the / will be backup, exclude /media , /.journal and /.fsck

FileSet {
  Name = "systemx2 Full Set"
  Include {
    Options {
      signature = MD5
      compression = GZIP
    }
    File = /
  }

  Exclude {
    File = /media
    File = /.journal
    File = /.fsck
  }
}


# List of files to be backed up for pseries

FileSet {
  Name = "pseries Full Set"
  Include {
    Options {
      signature = MD5
      compression = GZIP
    }
    File = /
  }

  Exclude {
    File = /media
    File = /.journal
    File = /.fsck
  }
}


# When to do the backups, full backup on first sunday of the month,
#  differential (i.e. incremental since full) every other sunday,
#  and incremental backups other days
Schedule {
  Name = "WeeklyCycle"
  Run = Full 1st sun at 23:05
  Run = Differential 2nd-5th sun at 23:05
  Run = Incremental mon-sat at 23:05
}


# This schedule does the catalog. It starts after the WeeklyCycle
Schedule {
  Name = "WeeklyCycleAfterBackup"
  Run = Full sun-sat at 23:10
}

# This is the backup of the catalog
FileSet {
  Name = "Catalog"
  Include {
    Options {
      signature = MD5
    }
    File = "/usr/local/bacula//bacula.sql"
  }
}


# CLIENT Definition, this reflect what you have configured above. IP address of the client is set here.

# localhost

Client {
  Name = p770vlpar3-fd
  Address = p770vlpar3
  FDPort = 9102
  Catalog = MyCatalog
  Password = "oscc123"          # password for FileDaemon
  File Retention = 30 days            # 30 days
  Job Retention = 6 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}

# systemx2

Client {
  Name = systemx2
  Address = 172.23.136.38
  FDPort = 9102
  Catalog = MyCatalog
  Password = "oscc123"          # password for FileDaemon
  File Retention = 30 days            # 30 days
  Job Retention = 6 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}

# pseries

Client {
  Name = pseries
  Address = 172.23.136.53
  FDPort = 9102
  Catalog = MyCatalog
  Password = "oscc123"          # password for FileDaemon
  File Retention = 30 days            # 30 days
  Job Retention = 6 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}


# Definition of tape storage device
Storage {
  Name = ibmts3200
  Address = p770vlpar3               
  SDPort = 9103
  Password = "oscc123"
  Device = ibmts3200
  Media Type = LTO-2
  Autochanger = yes

}


# Generic catalog service
Catalog {
  Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:mysql"; dbaddress = 127.0.0.1; dbport = 
  dbname = "bacula"; dbuser = "bacula"; dbpassword = ""
}

# Reasonable message delivery -- send most everything to email address
#  and to the console
Messages {
  Name = Standard
  mailcommand = "/usr/local/bacula/bin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/usr/local/bacula/bin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervention needed for %j\" %r"
  mail = root@localhost = all, !skipped           
  operator = root@localhost = mount
  console = all, !skipped, !saved
  append = "/usr/local/bacula//log" = all, !skipped
  catalog = all
}


# Message delivery for daemon messages (no job).
Messages {
  Name = Daemon
  mailcommand = "/usr/local/bacula/bin/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message\" %r"
  mail = root@localhost = all, !skipped           
  console = all, !skipped, !saved
  append = "/usr/local/bacula//log" = all, !skipped
}

# Oracle pool for testing, define a pool here
Pool {
  Name = oracle
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365 days         # one year
}

# File Pool definition
#Pool {
#  Name = File
#  Pool Type = Backup
#  Recycle = yes                       # Bacula can automatically recycle Volumes
#  AutoPrune = yes                     # Prune expired volumes
#  Volume Retention = 365 days         # one year
#  Maximum Volume Bytes = 50G          # Limit Volume size to something reasonable
#  Maximum Volumes = 100               # Limit number of Volumes in Pool
#}


# Scratch pool definition
#Pool {
#  Name = Scratch
#  Pool Type = Backup
#}

#
# Restricted console used by tray-monitor to get the status of the director
# Define a password for bacula console here, you opt to use the text based console or BAT GUI

Console {
  Name = p770vlpar3-mon
  Password = "oscc123"
  CommandACL = status, .status
}

#Done

save and exit.

restart bacula.

/usr/local/bacula/bin/bacula stop

/usr/local/bacula/bin/bacula start

Look for error message on the console, you may have typo in the configuration files.

We have done with the server configuration and now lets move to client installation and configuration.

Console configuration

You use bacula console to administer backup activities, this also include the initial configuration such as tape labeling, list client etc. You will specify password to connect to the bacula director as previous.

Modify bacula-console.conf as below:

#
# Bacula User Agent (or Console) Configuration File
#

Director {
  Name = p770vlpar3-dir
  DIRport = 9101
  address = p770vlpar3
  Password = "oscc123"
}

#End

Client Installation and configuration

Using the same package as for the bacula server with the following:

In the bacula source directory, on the client (Unix/Linux).

./configure --sbindir=/usr/local/bacula/bin --sysconfdir=/usr/local/bacula/bin --with-pid-dir=/usr/local/bacula/bin/working --with-subsysdir=/usr/local/bacula/bin/working --with-working-dir=/usr/local/bacula/ --enable-client-only

make
make install
make install-autostart

Modify the bacula-fd.conf on the client as below:

#
# Default  Bacula File Daemon Configuration file
#
#  For Bacula release 5.0.3 (04 August 2010) -- suse 11
#
# There is not much to change here except perhaps the
# File daemon Name to
#

#
# List Directors who are permitted to contact this File daemon
#
Director {
  Name = p770vlpar3-dir
  Password = "oscc123"
}

#
# Restricted Director, used by tray-monitor to get the
#   status of the file daemon
#
Director {
  Name = p770vlpar3-mon
  Password = "oscc123"
  Monitor = yes
}

#
# "Global" File daemon configuration specifications
#
FileDaemon {                          # this is me
  Name = p770vlpar3-fd
  FDport = 9102                  # where we listen for the director
  WorkingDirectory = /usr/local/bacula/
  Pid Directory = /usr/local/bacula/bin/working
  Maximum Concurrent Jobs = 20
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = p770vlpar3-dir = all, !skipped, !restored
}

#Done

Save and exit from the file.

service bacula-fd restart

or

/usr/local/bacula/bin/bacula-fd start

Provided that you have configured your firewall to allow traffics from and to the bacula server, you may now have working configuration and test can be made accordingly.