Production Server Storage Replacement & Live Data Migration SOP

1-Line Summary

Replacing production server storage requires (1) Write transaction freezing for integrity, (2) rsync -avAX ownership/SELinux preservation, (3) Persistent /etc/fstab UUID binding, and (4) Non-destructive rename rollbacks.

1. 4 Core Principles of Zero-Data-Loss Migration

[Zero-Data-Loss Directory Migration Workflow]
1. Baseline Sync (rsync #1) ──(Online System)──> [95% data transferred in background]
2. Daemon Shutdown (Stop)    ──(Maintenance Window)──> [Zero write transactions]
3. Delta Sync (rsync #2)    ──(Takes seconds)──> [100% consistent state]
4. Mount Swap & Startup     ──(/etc/fstab UUID)──> [Workloads resumed]

---

2. Step-by-Step SOP

Step 1. Online Background Pre-Sync

Transfer bulk data while the application runs to minimize downtime:

# Preserve archive ownership (-a), ACLs (-A), and SELinux contexts (-X)
rsync -avAX --progress /var/lib/mysql/ /data/mysql_new/

Step 2. Clean Service Daemon Shutdown

Flush dirty buffer pools and transaction write-ahead logs:

systemctl stop mariadb

Step 3. Final Delta Synchronization

rsync -avAX --delete /var/lib/mysql/ /data/mysql_new/
ls -ld /data/mysql_new

Step 4. Persistent /etc/fstab UUID Binding

Device names (/dev/sdb1) can reorder across reboots; always bind via UUID:

blkid /dev/sdb1
# /etc/fstab: UUID=xxxx-xxxx /data xfs defaults,noatime 0 0

Step 5. Safety Rename & Service Restart

mv /var/lib/mysql /var/lib/mysql_backup_20260517
ln -s /data/mysql_new /var/lib/mysql
systemctl start mariadb

---

3. Gotchas & Engineering Checkpoints

1. SELinux Permission Denied: If -X was omitted, MariaDB cannot access new storage paths; restore security labels via restorecon -R /data/mysql_new.
2. noatime Optimization: Mount high-write database filesystems with noatime to eliminate metadata update write overheads.

---

Published: 2026-05-17 06:09:18Updated: 2026-08-15 13:57:00