feat: complete LVM backup system with external M.2 boot support
MAJOR MILESTONE: Transform backup system into comprehensive LVM migration solution 🎯 LVM Migration & Boot System Complete: - Complete external M.2 LVM migration capability - One-button migration from non-LVM to LVM system - Automatic GRUB repair and boot configuration - External boot validation and recovery tools 🔧 New Migration Tools Added: - fix_grub_lvm_boot.sh: Complete GRUB repair for external LVM boot - automated_clonezilla_backup.sh: Automated backup with Clonezilla integration - validate_lvm_migration.sh: Comprehensive migration validation - troubleshoot_migration.sh: Advanced diagnostic and repair tools - emergency_install.sh: Package installation for live systems - bootstrap_usb_tools.sh: USB preparation with all dependencies 💾 Backup System Enhancements: - create_alpine_backup_usb.sh: Alpine Linux live system preparation - create_clonezilla_backup.sh: Professional backup solution integration - plug_and_play_backup.sh: Simple automated backup workflow - lvm_snapshot_backup.sh: LVM snapshot-based incremental backups - simple_auto_backup.sh: Streamlined backup automation 📋 Documentation & Guides: - LIVE_USB_MIGRATION_GUIDE.md: Complete migration walkthrough - DRIVE_SELECTION_REFERENCE.md: Safe drive selection procedures - Comprehensive troubleshooting and validation procedures - Step-by-step migration instructions with safety checks 🛡️ Safety & Validation Features: - Interactive drive selection with confirmation - Comprehensive pre-migration checks - Automatic backup validation - GRUB boot repair with fallback options - Hardware compatibility verification 🧪 Testing & Debugging: - Complete GRUB configuration analysis - LVM volume validation and repair - Boot sequence troubleshooting - Hardware connection diagnostics ✅ Production Ready Status: - All migration tools tested and validated - External M.2 boot functionality confirmed - GRUB configuration properly generates LVM entries - Kernel files correctly deployed to external boot partition - EFI bootloader properly configured as 'ubuntu-external' This completes the transformation from simple backup scripts to a comprehensive LVM migration and backup system capable of full system migration to external M.2 with proper boot configuration and recovery capabilities.
This commit is contained in:
root
515
LIVE_USB_MIGRATION_GUIDE.md
Normal file
515
LIVE_USB_MIGRATION_GUIDE.md
Normal file
@@ -0,0 +1,515 @@
|
||||
# LVM Migration Guide: Live USB to LVM System
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides comprehensive instructions for migrating your current non-LVM system to an LVM-based system on an external M.2 SSD. **This migration MUST be performed from a live USB system** to avoid file system conflicts and ensure data integrity.
|
||||
|
||||
## Why Migrate to LVM?
|
||||
|
||||
**Benefits of LVM System:**
|
||||
- **Instant Snapshots**: Create consistent backups without downtime
|
||||
- **Flexible Storage**: Resize volumes dynamically without repartitioning
|
||||
- **Advanced Backups**: Snapshot-based backups with rollback capability
|
||||
- **Space Efficiency**: Snapshots only store changes, not full copies
|
||||
- **System Recovery**: Easy rollback to previous states
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Hardware Requirements
|
||||
- **Live USB System**: Boot from any Linux live USB (Ubuntu, Debian, etc.)
|
||||
- **Internal Drive**: Your current system (will remain unchanged)
|
||||
- **External M.2 SSD**: Target drive for LVM system (will be formatted)
|
||||
- **Sufficient Space**: External drive should be ≥ size of used space on internal drive
|
||||
|
||||
### Software Requirements
|
||||
- Live USB system with LVM tools (will be installed automatically)
|
||||
- Network connection for package installation
|
||||
- Root/sudo access on live system
|
||||
|
||||
## Before You Begin
|
||||
|
||||
### 1. Create Recovery Environment
|
||||
```bash
|
||||
# Prepare live USB with migration tools
|
||||
# Download latest Ubuntu/Debian live ISO
|
||||
# Flash to USB drive using dd or balenaEtcher
|
||||
```
|
||||
|
||||
### 2. Backup Important Data
|
||||
⚠️ **CRITICAL**: While the internal drive remains unchanged, create an additional backup of critical data before proceeding.
|
||||
|
||||
### 3. Document Current System
|
||||
```bash
|
||||
# Boot your current system and document the configuration
|
||||
lsblk -f > system_layout.txt
|
||||
df -h > disk_usage.txt
|
||||
cat /etc/fstab > fstab_backup.txt
|
||||
```
|
||||
|
||||
## Migration Process
|
||||
|
||||
### Step 1: Boot from Live USB System
|
||||
|
||||
1. **Shutdown your system** completely
|
||||
2. **Insert live USB** and connect external M.2 SSD
|
||||
3. **Configure BIOS/UEFI**:
|
||||
- Set USB as first boot device
|
||||
- Ensure both internal and external drives are detected
|
||||
4. **Boot live system**:
|
||||
- Select "Try Ubuntu" or "Live System" (not "Install")
|
||||
- Wait for desktop to load completely
|
||||
5. **Open terminal** and gain root access:
|
||||
```bash
|
||||
sudo -i
|
||||
```
|
||||
|
||||
### Step 2: Prepare Live System
|
||||
|
||||
```bash
|
||||
# Download and prepare the migration tools
|
||||
cd /tmp
|
||||
git clone <your-repo> backup_tools
|
||||
cd backup_tools
|
||||
|
||||
# Or if you have the tools on external drive already:
|
||||
mkdir -p /mnt/temp
|
||||
mount /dev/sda1 /mnt/temp # Adjust device as needed
|
||||
cp -r /mnt/temp/migration_tools/* /tmp/
|
||||
umount /mnt/temp
|
||||
|
||||
# Prepare the live system
|
||||
./prepare_live_system.sh
|
||||
```
|
||||
|
||||
**This script will:**
|
||||
- ✅ Verify you're running from live system
|
||||
- 📦 Install required packages (lvm2, cryptsetup, rsync, etc.)
|
||||
- 🔧 Load kernel modules for LVM
|
||||
- 💽 Detect available drives
|
||||
- 📁 Create migration workspace
|
||||
|
||||
### Step 3: Run Migration Script
|
||||
|
||||
```bash
|
||||
# Execute the migration (this will take 30-90 minutes)
|
||||
./migrate_to_lvm.sh
|
||||
```
|
||||
|
||||
**The migration process includes:**
|
||||
|
||||
1. **Drive Detection** (Automatic):
|
||||
```
|
||||
Detecting drives...
|
||||
Available drives:
|
||||
1. /dev/nvme0n1 - 477GB Samsung SSD 980 (Internal)
|
||||
2. /dev/sda - 477GB Samsung T7 (External USB)
|
||||
|
||||
Selected drives:
|
||||
Internal (source): /dev/nvme0n1
|
||||
External (target): /dev/sda
|
||||
```
|
||||
|
||||
2. **System Analysis**:
|
||||
- Automatically detects partition layout
|
||||
- Identifies filesystem types
|
||||
- Handles encrypted partitions
|
||||
- Calculates optimal LVM sizes
|
||||
|
||||
3. **Confirmation Prompts**:
|
||||
```
|
||||
⚠️ WARNING: This will DESTROY all data on /dev/sda!
|
||||
|
||||
Migration Summary:
|
||||
Source: /dev/nvme0n1 (non-LVM system)
|
||||
Target: /dev/sda (will become LVM system)
|
||||
Root size: 70G
|
||||
Home size: 350G
|
||||
Swap size: 16G
|
||||
Boot size: 2G
|
||||
|
||||
Do you want to continue? [y/N]
|
||||
```
|
||||
|
||||
4. **LVM Layout Creation**:
|
||||
- Creates GPT partition table
|
||||
- EFI boot partition (512MB)
|
||||
- LVM physical volume (remaining space)
|
||||
- Creates volume group and logical volumes
|
||||
|
||||
5. **Data Migration**:
|
||||
- Mounts source filesystems (handles encryption)
|
||||
- Copies all system data with rsync
|
||||
- Preserves permissions, links, and attributes
|
||||
- Updates system configuration files
|
||||
|
||||
6. **System Configuration**:
|
||||
- Updates /etc/fstab for LVM volumes
|
||||
- Configures initramfs for LVM support
|
||||
- Installs and configures GRUB bootloader
|
||||
- Creates LVM snapshot backup tools
|
||||
|
||||
### Step 4: Validation and Testing
|
||||
|
||||
```bash
|
||||
# Validate the migration
|
||||
./validate_lvm_migration.sh
|
||||
```
|
||||
|
||||
**Validation checks:**
|
||||
- ✅ LVM volumes created correctly
|
||||
- ✅ Filesystems are healthy
|
||||
- ✅ Boot configuration is valid
|
||||
- ✅ GRUB installation successful
|
||||
- ✅ System files copied completely
|
||||
- ✅ LVM snapshot capability working
|
||||
|
||||
### Step 5: First Boot Test
|
||||
|
||||
1. **Cleanup and shutdown**:
|
||||
```bash
|
||||
# Clean up and prepare for reboot
|
||||
sync
|
||||
umount -a
|
||||
shutdown -h now
|
||||
```
|
||||
|
||||
2. **Configure BIOS/UEFI**:
|
||||
- Boot into BIOS/UEFI settings
|
||||
- Change boot order: External M.2 SSD as first boot device
|
||||
- Save and exit
|
||||
|
||||
3. **Test boot from external M.2**:
|
||||
- System should boot normally from external drive
|
||||
- Login and verify everything works
|
||||
- Check that all your files and settings are present
|
||||
|
||||
4. **Verify LVM system**:
|
||||
```bash
|
||||
# Check LVM status
|
||||
sudo lvs
|
||||
sudo vgs
|
||||
sudo pvs
|
||||
|
||||
# Check filesystem mounts
|
||||
df -h
|
||||
cat /proc/mounts | grep mapper
|
||||
```
|
||||
|
||||
## System Configuration Details
|
||||
|
||||
### LVM Layout Created
|
||||
```
|
||||
Physical Volume: /dev/sda2
|
||||
Volume Group: system-vg
|
||||
Logical Volumes:
|
||||
├── root (70G) - ext4 - mounted at /
|
||||
├── home (350G) - ext4 - mounted at /home
|
||||
├── boot (2G) - ext4 - mounted at /boot
|
||||
└── swap (16G) - swap - swap space
|
||||
|
||||
Additional:
|
||||
├── /dev/sda1 (512M) - vfat - EFI boot partition - mounted at /boot/efi
|
||||
└── Free space (~38G) - available for snapshots and volume expansion
|
||||
```
|
||||
|
||||
### Migration Advantages
|
||||
|
||||
**Flexibility:**
|
||||
- Resize any volume without repartitioning
|
||||
- Add new drives to volume group
|
||||
- Move logical volumes between physical drives
|
||||
|
||||
**Backup & Recovery:**
|
||||
- Create instant snapshots of any volume
|
||||
- Rollback changes using snapshots
|
||||
- Consistent backups without downtime
|
||||
|
||||
**Space Management:**
|
||||
- Thin provisioning support
|
||||
- Automatic space allocation
|
||||
- Easy expansion and shrinking
|
||||
|
||||
## Using LVM Snapshots
|
||||
|
||||
### Basic Snapshot Operations
|
||||
|
||||
```bash
|
||||
# Create snapshots for backup
|
||||
sudo ./lvm_snapshot_backup.sh backup
|
||||
|
||||
# Snapshots are mounted at:
|
||||
/mnt/backup/root # Snapshot of root filesystem
|
||||
/mnt/backup/home # Snapshot of home filesystem
|
||||
/mnt/backup/boot # Snapshot of boot filesystem
|
||||
|
||||
# Perform backup to external storage
|
||||
rsync -avH /mnt/backup/ /path/to/external/backup/
|
||||
|
||||
# Clean up snapshots
|
||||
sudo ./lvm_snapshot_backup.sh remove
|
||||
```
|
||||
|
||||
### Advanced LVM Operations
|
||||
|
||||
```bash
|
||||
# Extend a logical volume (add 10GB to home)
|
||||
sudo lvextend -L +10G /dev/system-vg/home
|
||||
sudo resize2fs /dev/system-vg/home
|
||||
|
||||
# Create additional logical volume
|
||||
sudo lvcreate -L 20G -n data system-vg
|
||||
sudo mkfs.ext4 /dev/system-vg/data
|
||||
sudo mkdir /data
|
||||
sudo mount /dev/system-vg/data /data
|
||||
|
||||
# Snapshot before system changes
|
||||
sudo lvcreate -L 5G -s -n root-before-update /dev/system-vg/root
|
||||
|
||||
# Rollback if needed
|
||||
sudo umount /
|
||||
sudo lvconvert --merge /dev/system-vg/root-before-update
|
||||
# Reboot to activate rollback
|
||||
```
|
||||
|
||||
## Troubleshooting Guide
|
||||
|
||||
### Migration Issues
|
||||
|
||||
**Migration Script Fails**
|
||||
```bash
|
||||
# Check logs for detailed error information
|
||||
tail -f /var/log/lvm-migration.log
|
||||
|
||||
# Common issues and solutions:
|
||||
```
|
||||
|
||||
| Issue | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| "Drive not found" | Drive not connected/detected | Check connections, try different USB port |
|
||||
| "Insufficient space" | Target drive too small | Use larger drive or reduce partition sizes |
|
||||
| "LVM tools not found" | Missing packages | Run `prepare_live_system.sh` first |
|
||||
| "Permission denied" | Not running as root | Use `sudo` or `sudo -i` |
|
||||
| "Mount failed" | Filesystem corruption | Check drive with `fsck` |
|
||||
|
||||
**Encrypted Partition Issues**
|
||||
```bash
|
||||
# If encrypted partition unlock fails:
|
||||
sudo cryptsetup luksOpen /dev/nvme0n1p3 temp-unlock
|
||||
# Enter correct password
|
||||
sudo cryptsetup close temp-unlock
|
||||
```
|
||||
|
||||
**Drive Detection Problems**
|
||||
```bash
|
||||
# Manually check drives
|
||||
lsblk -dpno NAME,SIZE,MODEL
|
||||
sudo fdisk -l
|
||||
|
||||
# If drives not detected:
|
||||
sudo partprobe # Re-read partition tables
|
||||
sudo udevadm settle # Wait for device detection
|
||||
```
|
||||
|
||||
### Boot Issues After Migration
|
||||
|
||||
**System Won't Boot from External Drive**
|
||||
|
||||
1. **Check BIOS/UEFI Settings**:
|
||||
- Verify external M.2 is detected in BIOS
|
||||
- Set correct boot priority
|
||||
- Enable UEFI boot mode
|
||||
- Disable Secure Boot if necessary
|
||||
|
||||
2. **Repair GRUB from Live USB**:
|
||||
```bash
|
||||
# Boot from live USB and mount LVM system
|
||||
sudo vgchange -ay system-vg
|
||||
sudo mount /dev/system-vg/root /mnt
|
||||
sudo mount /dev/system-vg/boot /mnt/boot
|
||||
sudo mount /dev/sda1 /mnt/boot/efi
|
||||
|
||||
# Reinstall GRUB
|
||||
sudo mount --bind /dev /mnt/dev
|
||||
sudo mount --bind /proc /mnt/proc
|
||||
sudo mount --bind /sys /mnt/sys
|
||||
sudo chroot /mnt
|
||||
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian
|
||||
update-grub
|
||||
exit
|
||||
|
||||
# Cleanup and reboot
|
||||
sudo umount /mnt/dev /mnt/proc /mnt/sys
|
||||
sudo umount /mnt/boot/efi /mnt/boot /mnt
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
**Emergency Recovery**
|
||||
|
||||
If external system is completely broken:
|
||||
1. Change BIOS boot order back to internal drive
|
||||
2. Boot from original system (unchanged)
|
||||
3. Re-attempt migration or continue with original system
|
||||
|
||||
### LVM Issues
|
||||
|
||||
**Volume Group Not Found**
|
||||
```bash
|
||||
# Activate volume group manually
|
||||
sudo vgchange -ay system-vg
|
||||
|
||||
# Scan for volume groups
|
||||
sudo vgscan
|
||||
sudo pvscan
|
||||
```
|
||||
|
||||
**Snapshot Issues**
|
||||
```bash
|
||||
# Remove stuck snapshots
|
||||
sudo umount /mnt/backup/root /mnt/backup/home 2>/dev/null || true
|
||||
sudo lvremove -f system-vg/root-snapshot
|
||||
sudo lvremove -f system-vg/home-snapshot
|
||||
|
||||
# Check volume group free space
|
||||
sudo vgs system-vg
|
||||
```
|
||||
|
||||
**File System Corruption**
|
||||
```bash
|
||||
# Check and repair LVM volumes
|
||||
sudo fsck /dev/system-vg/root
|
||||
sudo fsck /dev/system-vg/home
|
||||
sudo fsck /dev/system-vg/boot
|
||||
```
|
||||
|
||||
## Recovery Procedures
|
||||
|
||||
### Complete Rollback to Original System
|
||||
|
||||
If you decide to abandon LVM migration:
|
||||
|
||||
1. **Boot from internal drive**:
|
||||
- Change BIOS boot order to internal drive
|
||||
- Boot normally from original system
|
||||
|
||||
2. **Reformat external drive** (optional):
|
||||
```bash
|
||||
# Wipe LVM configuration
|
||||
sudo dd if=/dev/zero of=/dev/sda bs=1M count=100
|
||||
# Or use backup tools to restore external drive
|
||||
```
|
||||
|
||||
3. **Continue with original system**:
|
||||
- Everything remains as before migration
|
||||
- Use existing backup tools for regular backups
|
||||
|
||||
### Retry Migration
|
||||
|
||||
If you want to attempt migration again:
|
||||
|
||||
1. **Boot from live USB**
|
||||
2. **Run migration script again**:
|
||||
```bash
|
||||
./migrate_to_lvm.sh
|
||||
```
|
||||
- Script will destroy existing LVM setup and recreate
|
||||
- Source system (internal) remains unchanged
|
||||
|
||||
### Disaster Recovery
|
||||
|
||||
**If both systems fail:**
|
||||
|
||||
1. **Boot from live USB**
|
||||
2. **Mount internal drive** for data recovery:
|
||||
```bash
|
||||
mkdir -p /mnt/recovery
|
||||
|
||||
# Mount root partition
|
||||
sudo mount /dev/nvme0n1p1 /mnt/recovery
|
||||
|
||||
# If home is encrypted:
|
||||
sudo cryptsetup open /dev/nvme0n1p3 recovery-home
|
||||
sudo mkdir -p /mnt/recovery/home
|
||||
sudo mount /dev/mapper/recovery-home /mnt/recovery/home
|
||||
|
||||
# Copy important data to external storage
|
||||
rsync -avH /mnt/recovery/home/username/ /path/to/safe/backup/
|
||||
```
|
||||
|
||||
3. **Fresh OS installation** if needed:
|
||||
- Install fresh OS on any drive
|
||||
- Restore personal data from backup
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### LVM Performance Tuning
|
||||
|
||||
```bash
|
||||
# Enable read-ahead for better performance
|
||||
sudo blockdev --setra 2048 /dev/system-vg/root
|
||||
sudo blockdev --setra 2048 /dev/system-vg/home
|
||||
|
||||
# Add to /etc/fstab for persistent read-ahead:
|
||||
# /dev/system-vg/root / ext4 defaults,noatime 0 1
|
||||
# /dev/system-vg/home /home ext4 defaults,noatime 0 2
|
||||
```
|
||||
|
||||
### Snapshot Management
|
||||
|
||||
```bash
|
||||
# Monitor snapshot usage
|
||||
sudo lvs -a -o lv_name,lv_size,data_percent system-vg
|
||||
|
||||
# Remove old snapshots regularly
|
||||
sudo lvremove system-vg/old-snapshot-name
|
||||
|
||||
# Set up automatic snapshot cleanup (cron job)
|
||||
echo '0 2 * * * root /usr/local/bin/lvm-snapshot-backup.sh remove' >> /etc/crontab
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Regular Maintenance
|
||||
|
||||
1. **Monitor disk space**:
|
||||
```bash
|
||||
sudo vgs system-vg # Check volume group free space
|
||||
df -h # Check filesystem usage
|
||||
```
|
||||
|
||||
2. **Regular snapshots**:
|
||||
```bash
|
||||
# Before system updates
|
||||
sudo lvcreate -L 5G -s -n pre-update-$(date +%Y%m%d) /dev/system-vg/root
|
||||
|
||||
# Before major changes
|
||||
sudo ./lvm_snapshot_backup.sh backup
|
||||
```
|
||||
|
||||
3. **Backup strategy**:
|
||||
- Daily: LVM snapshots to external storage
|
||||
- Weekly: Full system backup using existing tools
|
||||
- Monthly: Verify backup integrity
|
||||
|
||||
### Security Considerations
|
||||
|
||||
- **Encryption**: Home data is no longer encrypted in LVM setup
|
||||
- Consider full disk encryption if security is critical
|
||||
- Use file-level encryption for sensitive data
|
||||
|
||||
- **Access Control**: Secure LVM management commands
|
||||
```bash
|
||||
# Restrict LVM command access
|
||||
sudo chmod 750 /usr/local/bin/lvm-*
|
||||
```
|
||||
|
||||
## Summary
|
||||
|
||||
The migration successfully transforms your system from traditional partitions to a flexible LVM-based setup, providing:
|
||||
|
||||
✅ **Instant snapshots** for consistent backups
|
||||
✅ **Dynamic volume resizing** without downtime
|
||||
✅ **Advanced backup strategies** with rollback capability
|
||||
✅ **Space efficiency** with thin provisioning
|
||||
✅ **System recovery** options with snapshots
|
||||
|
||||
Your original system remains intact as a fallback, making this a low-risk enhancement to your backup and storage capabilities.
|
||||
Reference in New Issue
Block a user