homelab-migration

Migrate LXCs and VMs between two Proxmox VE clusters with migrate-guest.sh. Run it as root on the source node; it auto-detects the guest type per VMID and drives pct remote-migrate / qm remote-migrate — the same mechanism Proxmox Datacenter Manager uses, but scriptable and with the container feature-flag problem handled automatically.

Why this is needed

Cross-cluster migration authenticates against the target with an API token (e.g. root@pam!migrate). Proxmox only allows the actual root@pam user — never a token — to set LXC feature flags other than nesting. Most community-script containers ship with features: nesting=1,keyctl=1, so migrating them via Datacenter Manager or plain remote-migrate fails after the full disk copy with:

403 Permission check failed (changing feature flags (except nesting) is only allowed for root@pam)

The script works around this per container:

1. Stops the container if running (a feature change on a running CT only lands in [pve:pending] and the original flags would still be shipped).
2. Strips the features line down to nesting=1 (unprivileged CT) or removes it entirely (privileged CT — tokens can't set any features there).
3. Migrates with --restart --delete.
4. Restores the original features on the target over SSH (as real root, which is allowed to set them) and starts the container again.

On failure or Ctrl-C it restores the source container's features and running state, and every error path prints the exact manual command to recover.

HA-managed guests are handled automatically: remote-migrate refuses them outright, and the HA manager would restart a container behind the script's back when it shuts it down for the feature strip. The script removes the guest from HA (saving its settings) before doing anything else, re-adds it on the source if the migration fails, and on success reminds you to configure HA on the target cluster — HA groups don't transfer, so that part stays manual.

It also refuses up front — before any shutdown or disk transfer — guests that cross-cluster migration can't handle: bind mounts, device passthrough (dev0:), hookscripts, snapshots, and VM args: lines. Raw lxc.* config lines are silently dropped by Proxmox on the target, so the script prints them for you to re-add manually. Pool membership doesn't travel either; re-add guests to pools on the target by hand.

One-time setup

1. On the target cluster, create a migration token:

   pveum user token add root@pam migrate --privsep 0
   

2. Get the target node's certificate fingerprint (on the target):

   pvenode cert info | grep -A1 pve-ssl | grep Fingerprint
   

3. Set up SSH key auth from the source node to the target node, using the same address you put in TARGET_HOST (host keys are per-IP):

   ssh-copy-id root@<target>
   ssh root@<target>   # once, to accept the host key
   

4. Fill in the CONFIG block at the top of migrate-guest.sh: TARGET_HOST, TARGET_TOKEN, TARGET_FINGERPRINT, TARGET_STORAGE, TARGET_BRIDGE.

TARGET_HOST decides two things at once: which target node the guests land on, and which network path the data takes — all disk traffic flows to that address on port 8006. To move data over a fast link (e.g. a 40G LACP bond), give the bond an IP on both sides and use the target's bond IP here.

The script preflights everything it can before touching a guest: config placeholders, non-interactive SSH, and API reachability on port 8006.

Usage

./migrate-guest.sh <vmid> [vmid...]

Guests keep their VMID on the target cluster. Running VMs migrate live; stopped VMs migrate offline. Containers use restart-mode migration (brief downtime), and containers that need the feature strip arrive stopped and are started fresh on the target.

Test on one noncritical guest before running a batch.

Logging to a file

Pipe through tee to keep a record while still watching live progress. 2>&1 matters — the !! error/skip lines go to stderr and would otherwise be missing from the log:

./migrate-guest.sh 118 120 121 2>&1 | tee migration-$(date +%F).log

Run long batches inside tmux on the source node — if your SSH session drops mid-run, a plain shell HUPs the script and aborts the batch, while a tmux session just detaches and the migration keeps going:

tmux new -s migration
./migrate-guest.sh 118 120 121 2>&1 | tee migration-$(date +%F).log
# if disconnected, later:  tmux attach -t migration

Running migrations in parallel

The script migrates its arguments one at a time. To run migrations concurrently, start multiple instances with disjoint VMID lists (never the same VMID twice), each in its own tmux window/pane with its own log:

# window 1
./migrate-guest.sh 118 120 122 2>&1 | tee batch-a.log
# window 2
./migrate-guest.sh 121 123 125 2>&1 | tee batch-b.log

Parallel instances are safe — there's no shared state, and Proxmox's per-VMID locks catch accidental overlap. Two streams is a sensible ceiling: a single migration is one TCP flow and LACP hashes per-flow, so one transfer uses at most one bond member; beyond two you're mostly contending for storage I/O on both ends rather than moving data faster. Watch the first pair complete before launching more, so a config mistake fails once, not everywhere.