homelab-migration

Migrate LXCs and VMs between two Proxmox VE clusters with migrate-guest.sh. Run it as root on the source node; it auto-detects the guest type per VMID and drives pct remote-migrate / qm remote-migrate — the same mechanism Proxmox Datacenter Manager uses, but scriptable and with the container feature-flag problem handled automatically.

Why this is needed

Cross-cluster migration authenticates against the target with an API token (e.g. root@pam!migrate). Proxmox only allows the actual root@pam user — never a token — to set LXC feature flags other than nesting. Most community-script containers ship with features: nesting=1,keyctl=1, so migrating them via Datacenter Manager or plain remote-migrate fails after the full disk copy with:

403 Permission check failed (changing feature flags (except nesting) is only allowed for root@pam)

The script works around this per container:

1. Stops the container if running (a feature change on a running CT only lands in [pve:pending] and the original flags would still be shipped).
2. Strips the features line down to nesting=1 (unprivileged CT) or removes it entirely (privileged CT — tokens can't set any features there).
3. Migrates with --restart --delete.
4. Restores the original features on the target over SSH (as real root, which is allowed to set them) and starts the container again.

On failure or Ctrl-C it restores the source container's features and running state, and every error path prints the exact manual command to recover.

HA-managed guests are handled automatically: remote-migrate refuses them outright, and the HA manager would restart a container behind the script's back when it shuts it down for the feature strip. The script removes the guest from HA (saving its settings) before doing anything else, re-adds it on the source if the migration fails, and on success reminds you to configure HA on the target cluster — HA groups don't transfer, so that part stays manual.

Bind mounts on shared storage are handled the same way as the feature flags. A mount like

mp0: /mnt/pve/truenas-media,mp=/data,shared=1

isn't node-local data — /mnt/pve/<storeid> is where PVE mounts an NFS/CIFS storage, so the target cluster reaches the same share over the network and there is nothing to copy. Setting a bind mount is root-only, though, so the token-authenticated target rejects a config carrying one. The script strips the mpN: lines before migrating and re-adds them over SSH afterwards, inside the same rollback path as the features.

This only works if the target defines a storage with the same id — that's what makes /mnt/pve/<storeid> resolve to the same path on both sides:

pvesm add nfs truenas-media --server 10.10.10.10 --export /mnt/tank0/Media/plex --content images

The script checks the target's storages up front — before touching anything — and skips the guest, naming the storage, if the one it needs isn't there. Mounts without shared=1, or pointing outside /mnt/pve, are still refused: that data exists only on the source node and remote-migrate can't move it.

Two caveats on that check. It requires the storage to be active, not merely defined, so a target storage that exists but isn't currently mounted is skipped the same way a missing one is. And the storage list is fetched once per run and cached — if you bring a storage up on the target mid-batch, the running script won't see it, so re-run rather than expecting later guests to pick it up.

Both rest on assumptions worth knowing. /mnt/pve/<storeid> is the default path for an NFS/CIFS storage, not a guarantee — it can be overridden at creation time, which would break the same-id-same-path premise. And shared=1 is self-asserted: per pct.conf(5), "This option does not share the mount point automatically, it assumes it is shared already!" Neither is verified by PVE, so the script trusts what the source config declares. Both hold for storage created through the GUI.

It also refuses up front — before any shutdown or disk transfer — guests that cross-cluster migration can't handle: node-local bind mounts, device passthrough (dev0:), hookscripts, snapshots, and VM args: lines. Raw lxc.* config lines are silently dropped by Proxmox on the target, so the script prints them for you to re-add manually. Pool membership doesn't travel either; re-add guests to pools on the target by hand.

One-time setup

1. On the target cluster, create a migration token:

   pveum user token add root@pam migrate --privsep 0
   

2. Get the target node's certificate fingerprint (on the target):

   pvenode cert info | grep -A1 pve-ssl | grep Fingerprint
   

3. Set up SSH key auth from the source node to the target node, using the same address you put in TARGET_HOST (host keys are per-IP):

   ssh-copy-id root@<target>
   ssh root@<target>   # once, to accept the host key
   

4. Fill in the CONFIG block at the top of migrate-guest.sh: TARGET_HOST, TARGET_TOKEN, TARGET_FINGERPRINT, TARGET_STORAGE, TARGET_BRIDGE.

TARGET_HOST decides two things at once: which target node the guests land on, and which network path the data takes — all disk traffic flows to that address on port 8006. To move data over a fast link (e.g. a 40G LACP bond), give the bond an IP on both sides and use the target's bond IP here.

The script preflights everything it can before touching a guest: config placeholders, non-interactive SSH, and API reachability on port 8006.

The token ends up in the argument list of pct/qm remote-migrate, so it is visible in ps to any other local user for the duration of a migration. That's fine on a node only admins can log into — worth knowing if yours isn't.

If something fails, act on the printed commands before re-running

Every failure path prints the exact commands to put the guest right. Run them before retrying, because the script keeps no record of the original config between runs — it reads the live config as ground truth each time.

So if a feature or mount restore fails and you re-run without fixing it, the script sees the already-stripped config as the original and migrates the guest that way: the container arrives missing a mount or a feature flag, and nothing warns you, because as far as the second run can tell that's how it always was. This is the one failure mode the rollback logic cannot protect you from.

Usage

./migrate-guest.sh <vmid> [vmid...]

Guests keep their VMID on the target cluster. Running VMs migrate live; stopped VMs migrate offline. Containers use restart-mode migration (brief downtime), and containers that need a feature or bind-mount strip are stopped by the script, arrive stopped, and are started fresh on the target once their config has been restored.

Test on one noncritical guest before running a batch.

Logging to a file

Pipe through tee to keep a record while still watching live progress. 2>&1 matters — the !! error/skip lines go to stderr and would otherwise be missing from the log:

./migrate-guest.sh 118 120 121 2>&1 | tee migration-$(date +%F).log

Run long batches inside tmux on the source node — if your SSH session drops mid-run, a plain shell HUPs the script and aborts the batch, while a tmux session just detaches and the migration keeps going:

tmux new -s migration
./migrate-guest.sh 118 120 121 2>&1 | tee migration-$(date +%F).log
# if disconnected, later:  tmux attach -t migration

Running migrations in parallel

The script migrates its arguments one at a time. To run migrations concurrently, start multiple instances with disjoint VMID lists (never the same VMID twice), each in its own tmux window/pane with its own log:

# window 1
./migrate-guest.sh 118 120 122 2>&1 | tee batch-a.log
# window 2
./migrate-guest.sh 121 123 125 2>&1 | tee batch-b.log

Parallel instances are safe — there's no shared state, and Proxmox's per-VMID locks catch accidental overlap. Two streams is a sensible ceiling: a single migration is one TCP flow and LACP hashes per-flow, so one transfer uses at most one bond member; beyond two you're mostly contending for storage I/O on both ends rather than moving data faster. Watch the first pair complete before launching more, so a config mistake fails once, not everywhere.