root/backup_to_external_m.2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: Implement proper block-level Borg backups

Browse Source 
BREAKING CHANGE: Borg backups now store raw block devices instead of files

Changes:
- LV→Borg: Stores snapshot as raw block device (.img file) in Borg
- VG→Borg: Stores each LV as separate .img files in Borg repository
- No more file-level mounting - preserves exact block-level state
- Uses dd | borg create --stdin-name for LV backups
- Creates temporary .img files for VG backups
- Maintains all filesystem metadata, boot sectors, etc.
- Better deduplication for similar block patterns

Benefits:
- Exact block-level restoration possible
- Preserves all filesystem metadata
- Better suited for system/boot volume backups
- Still gets Borg's compression, deduplication, encryption
- Clear difference between LV and VG modes

Now LV→Borg and VG→Borg have distinct, useful purposes:
- LV→Borg: Single logical volume as one block device
- VG→Borg: All logical volumes as separate block devices
This commit is contained in:
root
2025-10-09 00:45:22 +02:00
parent 179a84e442
commit aee3d5019c
4 changed files with 113 additions and 137 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@@ -18,8 +18,8 @@ No complex migration logic, no fancy features that can break. Just reliable back
- **LV → LV**: Update existing logical volume backups
- **LV → Raw**: Create fresh backups on raw devices
- **VG → Raw**: Clone entire volume groups with LVM metadata
- **LV → Borg**: Backup logical volume to Borg repository (deduplicated, compressed, encrypted)
- **VG → Borg**: Backup entire volume group to Borg repository
- **LV → Borg**: Backup logical volume **block device** to Borg repository (preserves exact block-level state)
- **VG → Borg**: Backup **all block devices** from volume group to Borg repository
### Two Interfaces
@@ -55,15 +55,19 @@ sudo ./enhanced_simple_backup.sh vg-to-raw internal-vg /dev/sdb
sudo python3 simple_backup_gui.py
```
**Borg repository backups:**
**Borg repository backups (block-level):**
```bash
# Create new Borg repo and backup LV
# Create new Borg repo and backup LV as block device
sudo ./enhanced_simple_backup.sh lv-to-borg /dev/internal-vg/root /path/to/borg/repo --new-repo --encryption repokey --passphrase mypass
# Backup entire VG to existing repo
# Backup entire VG as multiple block devices to existing repo
sudo ./enhanced_simple_backup.sh vg-to-borg internal-vg /path/to/borg/repo --passphrase mypass
```
**Key Difference:**
- **LV → Borg**: Stores the raw snapshot as `{lv_name}.img` in Borg
- **VG → Borg**: Stores all LVs as separate `.img` files (`root.img`, `home.img`, etc.)
## Files
### Core System

49
enhanced_simple_backup.sh
View File

@@ -330,26 +330,18 @@ case "$MODE" in
log "Creating snapshot of source LV"
lvcreate -L1G -s -n "$SNAPSHOT_NAME" "$SOURCE" || error "Failed to create snapshot"
# Create temporary mount point
TEMP_MOUNT=$(mktemp -d -t borg_backup_XXXXXX)
# Mount snapshot
log "Mounting snapshot to $TEMP_MOUNT"
mount "$SNAPSHOT_PATH" "$TEMP_MOUNT" || error "Failed to mount snapshot"
# Create Borg archive
ARCHIVE_NAME="lv_${LV_NAME}_$(date +%Y%m%d_%H%M%S)"
log "Creating Borg archive: $ARCHIVE_NAME"
log "This may take a long time depending on data size..."
log "Creating Borg archive (block-level): $ARCHIVE_NAME"
log "Backing up raw snapshot block device to Borg..."
log "This preserves exact block-level state including filesystem metadata"
borg create --progress --stats "$TARGET::$ARCHIVE_NAME" "$TEMP_MOUNT" || error "Borg backup failed"
dd if="$SNAPSHOT_PATH" bs=4M | borg create --stdin-name "${LV_NAME}.img" --progress --stats "$TARGET::$ARCHIVE_NAME" - || error "Borg backup failed"
log "Borg backup completed successfully"
log "Block-level Borg backup completed successfully"
# Cleanup
log "Cleaning up mount point and snapshot"
umount "$TEMP_MOUNT" || warn "Failed to unmount"
rmdir "$TEMP_MOUNT"
log "Cleaning up snapshot"
lvremove -f "$SNAPSHOT_PATH" || warn "Failed to remove snapshot"
SNAPSHOT_PATH=""
@@ -400,16 +392,16 @@ case "$MODE" in
log "Found logical volumes: $(echo $LV_LIST | tr '\n' ' ')"
# Create base temp directory
# Create base temp directory for block images
BASE_TEMP_DIR=$(mktemp -d -t borg_vg_backup_XXXXXX)
SNAPSHOTS_CREATED=""
# Create snapshots and mount them
# Create snapshots and copy them to temporary block files
for LV_NAME in $LV_LIST; do
SNAPSHOT_NAME="${LV_NAME}_borg_snap"
SNAPSHOT_PATH="/dev/$SOURCE/$SNAPSHOT_NAME"
LV_PATH="/dev/$SOURCE/$LV_NAME"
MOUNT_POINT="$BASE_TEMP_DIR/$LV_NAME"
TEMP_IMAGE="$BASE_TEMP_DIR/${LV_NAME}.img"
log "Creating snapshot: $SNAPSHOT_NAME"
lvcreate -L500M -s -n "$SNAPSHOT_NAME" "$LV_PATH" || {
@@ -418,32 +410,25 @@ case "$MODE" in
}
SNAPSHOTS_CREATED="$SNAPSHOTS_CREATED $SNAPSHOT_PATH"
# Create mount point and mount
mkdir -p "$MOUNT_POINT"
log "Mounting $SNAPSHOT_PATH to $MOUNT_POINT"
mount "$SNAPSHOT_PATH" "$MOUNT_POINT" || {
warn "Failed to mount $SNAPSHOT_PATH"
# Copy snapshot to temporary image file
log "Creating block image for $LV_NAME"
dd if="$SNAPSHOT_PATH" of="$TEMP_IMAGE" bs=4M || {
warn "Failed to create block image for $LV_NAME"
continue
}
done
# Create Borg archive
ARCHIVE_NAME="vg_${SOURCE}_$(date +%Y%m%d_%H%M%S)"
log "Creating Borg archive: $ARCHIVE_NAME"
log "This may take a long time depending on data size..."
log "Creating Borg archive (block-level): $ARCHIVE_NAME"
log "Backing up all LV snapshots as raw block devices..."
borg create --progress --stats "$TARGET::$ARCHIVE_NAME" "$BASE_TEMP_DIR" || error "Borg backup failed"
log "Borg backup of entire VG completed successfully"
log "Block-level VG Borg backup completed successfully"
# Cleanup
log "Cleaning up mount points and snapshots"
for MOUNT_POINT in "$BASE_TEMP_DIR"/*; do
if mountpoint -q "$MOUNT_POINT" 2>/dev/null; then
umount "$MOUNT_POINT" || warn "Failed to unmount $MOUNT_POINT"
fi
done
log "Cleaning up temporary files and snapshots"
rm -rf "$BASE_TEMP_DIR"
for SNAPSHOT_PATH in $SNAPSHOTS_CREATED; do

10
list_drives.sh
View File

@@ -74,11 +74,13 @@ echo ""
echo "3. Entire VG to Raw Device (complete clone):"
echo " sudo ./enhanced_simple_backup.sh vg-to-raw internal-vg /dev/sdb"
echo ""
echo "4. LV to Borg Repository (encrypted backup):"
echo "4. LV to Borg Repository (block-level backup):"
echo " sudo ./enhanced_simple_backup.sh lv-to-borg /dev/internal-vg/root /path/to/borg/repo --new-repo"
echo " → Stores raw snapshot as 'root.img' in Borg repo"
echo ""
echo "5. Entire VG to Borg Repository (all LVs):"
echo "5. Entire VG to Borg Repository (all LVs as block devices):"
echo " sudo ./enhanced_simple_backup.sh vg-to-borg internal-vg /path/to/borg/repo --encryption repokey"
echo " → Stores each LV as separate .img files (root.img, home.img, etc.)"
echo ""
echo "=== GUI VERSION ==="
echo " sudo python3 simple_backup_gui.py"
@@ -88,7 +90,9 @@ echo "- Always run as root (sudo)"
echo "- lv-to-lv: Updates existing backup LV"
echo "- lv-to-raw: Creates fresh backup, overwrites target device"
echo "- vg-to-raw: Clones entire VG including LVM metadata"
echo "- lv-to-borg/vg-to-borg: Creates deduplicated, compressed, encrypted backups"
echo "- lv-to-borg/vg-to-borg: Creates block-level backups in Borg (preserves exact LV state)"
echo "- LV→Borg stores snapshot as single .img file, VG→Borg stores each LV as separate .img"
echo "- Borg backups are deduplicated, compressed, and encrypted"
echo "- Borg backups require: sudo apt install borgbackup"
echo "- Make sure target devices are unmounted before backup"
echo ""

177
simple_backup_gui.py
View File

@@ -490,8 +490,8 @@ class SimpleBackupGUI:
self.log("VG copy completed - target device now contains complete LVM structure")
def backup_lv_to_borg(self, source_lv, repo_path):
"""Backup LV to Borg repository"""
self.log("Mode: LV to Borg Repository")
"""Backup LV to Borg repository (block-level)"""
self.log("Mode: LV to Borg Repository (block-level backup)")
# Set up environment for Borg
borg_env = os.environ.copy()
@@ -523,57 +523,44 @@ class SimpleBackupGUI:
if not success:
raise Exception(f"Failed to create snapshot: {output}")
# Create a temporary mount point
import tempfile
temp_mount = tempfile.mkdtemp(prefix="borg_backup_")
# Create Borg backup of the raw block device
archive_name = f"lv_{lv_name}_{time.strftime('%Y%m%d_%H%M%S')}"
self.log(f"Creating Borg archive (block-level): {archive_name}")
self.log("Backing up raw snapshot block device to Borg...")
self.log("This preserves exact block-level state including filesystem metadata")
try:
# Mount the snapshot
self.log(f"Mounting snapshot to {temp_mount}")
success, output = self.run_command(f"mount {self.current_snapshot} {temp_mount}")
if not success:
raise Exception(f"Failed to mount snapshot: {output}")
# Create Borg backup
archive_name = f"lv_{lv_name}_{time.strftime('%Y%m%d_%H%M%S')}"
self.log(f"Creating Borg archive: {archive_name}")
self.log("This may take a long time depending on data size...")
borg_cmd = f"borg create --progress --stats {repo_path}::{archive_name} {temp_mount}"
# Run borg with environment
process = subprocess.Popen(borg_cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, env=borg_env)
# Stream output
for line in process.stdout:
# Use stdin mode to pipe the block device into borg
borg_cmd = f"dd if={self.current_snapshot} bs=4M | borg create --stdin-name '{lv_name}.img' --progress --stats {repo_path}::{archive_name} -"
# Run command with Borg environment
process = subprocess.Popen(borg_cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, env=borg_env)
# Stream output
for line in process.stdout:
if line.strip():
self.log(line.strip())
process.wait()
if process.returncode != 0:
raise Exception("Borg backup failed")
self.log("Borg backup completed successfully")
finally:
# Cleanup: unmount and remove temp directory
self.log("Cleaning up mount point")
self.run_command(f"umount {temp_mount}", show_output=False)
os.rmdir(temp_mount)
# Remove snapshot
self.log("Cleaning up snapshot")
success, output = self.run_command(f"lvremove -f {self.current_snapshot}")
if not success:
self.log(f"Warning: Failed to remove snapshot: {output}")
else:
self.log("Snapshot cleaned up")
self.current_snapshot = None
process.wait()
if process.returncode != 0:
raise Exception("Borg block-level backup failed")
self.log("Block-level Borg backup completed successfully")
# Cleanup snapshot
self.log("Cleaning up snapshot")
success, output = self.run_command(f"lvremove -f {self.current_snapshot}")
if not success:
self.log(f"Warning: Failed to remove snapshot: {output}")
else:
self.log("Snapshot cleaned up")
self.current_snapshot = None
def backup_vg_to_borg(self, source_vg, repo_path):
"""Backup entire VG to Borg repository"""
self.log("Mode: Entire VG to Borg Repository")
"""Backup entire VG to Borg repository (block-level)"""
self.log("Mode: Entire VG to Borg Repository (block-level backup)")
self.log("This will store all LV snapshots as raw block devices in Borg")
# Set up environment for Borg
borg_env = os.environ.copy()
@@ -606,7 +593,6 @@ class SimpleBackupGUI:
self.log(f"Found {len(lv_names)} logical volumes: {', '.join(lv_names)}")
snapshots_created = []
mount_points = []
try:
# Create snapshots for all LVs
@@ -620,62 +606,59 @@ class SimpleBackupGUI:
if not success:
raise Exception(f"Failed to create snapshot for {lv_name}: {output}")
snapshots_created.append(snapshot_path)
snapshots_created.append((snapshot_path, lv_name))
# Mount all snapshots
import tempfile
base_temp_dir = tempfile.mkdtemp(prefix="borg_vg_backup_")
for i, snapshot_path in enumerate(snapshots_created):
lv_name = lv_names[i]
mount_point = os.path.join(base_temp_dir, lv_name)
os.makedirs(mount_point)
mount_points.append(mount_point)
self.log(f"Mounting {snapshot_path} to {mount_point}")
success, output = self.run_command(f"mount {snapshot_path} {mount_point}")
if not success:
self.log(f"Warning: Failed to mount {snapshot_path}: {output}")
# Continue with other LVs
# Create Borg backup of the entire VG structure
# Create Borg archive with all block devices
archive_name = f"vg_{source_vg}_{time.strftime('%Y%m%d_%H%M%S')}"
self.log(f"Creating Borg archive: {archive_name}")
self.log("This may take a long time depending on data size...")
self.log(f"Creating Borg archive (block-level): {archive_name}")
self.log("Backing up all LV snapshots as raw block devices...")
borg_cmd = f"borg create --progress --stats {repo_path}::{archive_name} {base_temp_dir}"
# Create temporary directory structure for the archive
import tempfile
temp_dir = tempfile.mkdtemp(prefix="borg_vg_backup_")
# Run borg with environment
process = subprocess.Popen(borg_cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, env=borg_env)
# Stream output
for line in process.stdout:
self.log(line.strip())
process.wait()
if process.returncode != 0:
raise Exception("Borg backup failed")
self.log("Borg backup of entire VG completed successfully")
finally:
# Cleanup: unmount all mount points
for mount_point in mount_points:
if os.path.ismount(mount_point):
self.log(f"Unmounting {mount_point}")
self.run_command(f"umount {mount_point}", show_output=False)
# Remove temp directory
if 'base_temp_dir' in locals():
try:
# Copy all snapshots to temporary files that Borg can backup
for snapshot_path, lv_name in snapshots_created:
temp_file = os.path.join(temp_dir, f"{lv_name}.img")
self.log(f"Creating temporary image file for {lv_name}")
# Use dd to copy snapshot to temporary file
copy_cmd = f"dd if={snapshot_path} of={temp_file} bs=4M"
success, output = self.run_command(copy_cmd)
if not success:
raise Exception(f"Failed to create temp file for {lv_name}: {output}")
# Create Borg backup of all the block device images
borg_cmd = f"borg create --progress --stats {repo_path}::{archive_name} {temp_dir}"
# Run borg with environment
process = subprocess.Popen(borg_cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True, env=borg_env)
# Stream output
for line in process.stdout:
if line.strip():
self.log(line.strip())
process.wait()
if process.returncode != 0:
raise Exception("Borg VG backup failed")
self.log("Block-level VG Borg backup completed successfully")
finally:
# Remove temporary directory
import shutil
try:
shutil.rmtree(base_temp_dir)
shutil.rmtree(temp_dir)
self.log("Temporary files cleaned up")
except:
self.log(f"Warning: Could not remove temp directory {base_temp_dir}")
self.log(f"Warning: Could not remove temp directory {temp_dir}")
finally:
# Remove all snapshots
for snapshot_path in snapshots_created:
for snapshot_path, lv_name in snapshots_created:
self.log(f"Removing snapshot {snapshot_path}")
success, output = self.run_command(f"lvremove -f {snapshot_path}")
if not success: