ELK>> adlibre-backup>> 返回
项目作者： adlibre

项目描述 ：
High performance rsync backup utilising BTRFS / ZFS filesystem features
高级语言： Shell
项目地址: git://github.com/adlibre/adlibre-backup.git
创建时间： 2012-10-13T07:04:38Z
项目社区：https://github.com/adlibre/adlibre-backup
开源协议：Other
下载


Adlibre Backup

A high performance snapshot based backup system for Linux and UNIX like
operating systems.

Designed with system administrators in mind.

Utilises native BTRFS / ZFS filesystem features for seamless compression,
deduplication and snapshoting of the backup pool.

The problem

Existing Rsync backup approaches (eg Rsnapshot /
BackupPC) don’t scale, are hard to monitor
and maintain when used with dozens or hundreds of hosts.

They also don’t elegantly handle ad hoc backups, nor do they facilitate quickly
adding and removing hosts.

Our solution

Our solution is centralised and agentless, so there is nothing to consume
resources on your hosts, and all configuration is managed on the backup server.

Utilises BTRFS / ZFS native filesystem snapshots, and per host
filesystems for better performance, scalability and ease of management.

Backups can be run at anytime, with custom expiry and a short message so you
know why the backup was taken.

Aims

  • High performance and scalability. (Benchmarked faster than other Rsync backup
    solutions. eg BackupPC, Rsnapshot etc.)
  • Simplicity: Simple text based configuration. And simple files-on-disk
    backup format.
  • Aims to be a paranoid system administrator’s best friend.

Features

  • Agentless
  • Utilises BTRFS / ZFS filesystem features, eg snapshot, dedup
    and compression
  • Uses Rsync and
    SSH for transport
  • Integration with monitoring tools such as Nagios or Icinga using NSCA passive
    checks. Or easily integrate your own monitoring system by configuring a
    MONITOR_HANDLER script.
  • Centralised configuration and management - all configuration and scheduling is
    done on the backup server
  • Ad hoc annotated backups - allows for ad hoc backups with an explanation as to
    when or why the backup was taken and per backup retention periods
  • Per host backup, retention and quota policies
  • Per host configuration and logs stored with the snapshot
  • Utilise LVM snapshots for performing atomic backups of Linux systems. See
    atomic.sh (Work in progress).

Installation

High level installation is as follows:

  1. An operating system with BTRFS or ZFS support is required (eg
    FreeBSD or ZFS on Linux)
    and a dedicated storage pool.

  2. Check out the source code into the root of your /backup pool storage device
    and review ./conf/backup.conf. Modify as necessary to set your pool and
    filesystem options.

Detailed: Red Hat / CentOS / EL Installation and ZFS Usage Example

Create backup zpool with dedup and compression.

  1. zpool create -f backup vdb
  2. zfs set dedup=on backup
  3. zfs set compression=gzip backup

Install Adlibre Backup into root of backup zpool.

  1. yum -y install git
  2. cd /backup && git clone git://github.com/adlibre/adlibre-backup.git .

Install NSCA Client (optional) for Nagios / Icinga integration

  1. yum -y install nsca-client

Generate SSH Key, this is used for authentication.

  1. ssh-keygen -t rsa -N "" -f ~root/.ssh/id_rsa

Add server.example.com host config and copy the SSH Key to host example.com

  1. cd /backup && ./bin/add-host.sh example.com

Now run the backup

  1. ./bin/backup-runner.sh --all

The output

  1. [root@zbackup backup]# ./bin/backup-runner.sh --all
  2. Info: Begin backup run of hosts example.com
  3. Info: Begining backup of example.com
  4. Running: rsync -a --numeric-ids --hard-links --compress --delete-after --delete-excluded --fuzzy --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/tmp --exclude=/var/tmp --exclude=/var/run --exclude=/selinux --exclude=/cgroups --exclude=lost+found root@example.com:'/' /backup/hosts/example.com/d/
  5. Warning: NSCA Plugin not found.
  6. Backup Successful. Runtime 1757 seconds.
  7. Warning: NSCA Plugin not found.
  8. Snapshot example.com@2013-06-14-15:12:39-1371186759 Created
  9. Info: Completed backup of example.com

That’s it.

Now if you want to schedule daily backups Add the following to your root crontab:

  1. PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
  2. @daily /backup/bin/backup-runner.sh --all --comment "Backup Daily" && /backup/bin/prune.sh --all

Usage

Adding a host

./bin/add-host.sh <hostname>

Then customise the per host config in ./hosts/<hostname>/c/backup.conf and
ssh options in ~/.ssh/config if required.

Removing a host (ZFS)

To immediately purge the host configuration and all backup data:

zfs umount zfs-pool-name/hosts/<hostname> && zfs destroy zfs-pool-name/hosts/<hostname>

To disable future backups and allow existing backups to expire in line with the
retention policy set DISABLED=true in ./hosts/<hostname>/c/backup.conf.
This is the preferred method for host removal as it allows the old backups to
naturally expire.

Running an ad hoc backup of a single host

./bin/backup.sh <hostname> <annotation> <expiry-in-days>

Running a backup of all hosts

./bin/backup-runner.sh --all

or multiple hosts

./bin/backup-runner.sh <hostname> <hostname>...

Running scheduled backups with flexible retention

You can configure cron jobs to run regular backups with flexible retentions.

The following example keeps daily backups for 2 weeks, weekly backups for 6 months and monthly backups for a year.

  1. # daily backups, kept for 2 weeks
  2. 00 2 2-31 * * test $(date +\%u) != 7 && /backup/bin/backup-runner.sh --all --comment "Daily backup" --expiry 14 && /backup/bin/prune.sh --all
  3. # weekly backups, kept for 6 months
  4. 00 2 2-31 * * test $(date +\%u) = 7 && /backup/bin/backup-runner.sh --all --comment "Weekly backup" --expiry 180 && /backup/bin/prune.sh --all
  5. # monthly backups, kept for a year
  6. 00 2 1 * * /backup/bin/backup-runner.sh --all --comment "Monthly backup" --expiry 365 && /backup/bin/prune.sh --all

Restoring (ZFS)

All backups are stored on disk in plain sight. To restore all you need to do
is copy (or rsync) the files from the backup pool to your host.

To find a particular snapshot:

./bin/list-backups.sh <hostname>

eg:

  1. backup-host# ./bin/list-backups.sh example.com
  2. example.com 2012-10-25-23:35:19-1351168519 1352377190 successful "first backup"
  3. example.com 2012-11-04-15:40:49-1352004049 1354418267 successful "before acme software upgrade"

The files are stored plainly within the ZFS snapshot:

  1. backup-host# ls -lah /backup/hosts/example.com/.zfs/snapshot
  2. total 3
  3. dr-xr-xr-x  4 root  wheel     4B Nov 17 15:14 .
  4. dr-xr-xr-x  4 root  wheel     4B Oct 16 20:18 ..
  5. drwxr-xr-x  5 root  wheel     5B Oct 16 20:18 2012-10-25-23:35:19-1351168519
  6. drwxr-xr-x  5 root  wheel     5B Oct 16 20:18 2012-11-04-15:40:49-1352004049

Just dive in and copy the files out of the snapshot:

  1. cd /backup/hosts/example.com/.zfs/snapshot/ && \
  2. rsync -aH --numeric-ids 2012-11-04-15:40:49-1352004049/d/ example.com:/restore-point/

Upgrading

Upgrading is just a matter of updating the code and merging in any local changes
to the default configuration. If you’re using git to deploy the code then
git pull is usually sufficient.

However from time to time options and features do sometimes change. Please review
any changes to the defaults in etc/backup.conf.

The on disk pool format is stable and no changes are required to support upgrades.

Status

This has been in production use for many years now and is stable.

See TODO and ISSUES for outstanding issues.
And NOTES for development information.