VMware to Proxmox VE Migration — Complete Technical Tutorial

100%

GRIMOIRE❦

Grimoire Dindon Corpus Synthesis Volumes The Foundation of Iron

OPS

TECHNICAL TUTORIAL GUIDE · JUNE 2026

◆◆◆

VMWARE TO
PROXMOX VE MIGRATION
Complete Execution Tutorial · 13 Modules
Stretched Cluster D1 · D2 · Witness D3

◆

From initial audit to continuous operations
Commands, configuration files and validation checklists

◆◆◆

13Execution modules

3Datacenters D1/D2/D3

60+Commands & configs

4Failure scenarios

◆

TECHNICAL FOLLOW-UP — COMPLEMENT TO THE ARCHITECTURE STUDY

This guide is the operational follow-up to the architecture study. Where the study formalises technical choices and the target trajectory, this tutorial provides the commands, configuration files and checklists required to execute every layer of the migration, from initial audit through to continuous operations.

◆◆◆

All commands and configurations are generic and illustrative.
Adapt to your real environment before executing in production. Public document · CC BY-NC-SA 4.0

◆

Amine RAITI · Infrastructure Architect & SRE
Public document · CC BY-NC-SA 4.0 · AI Powered by Amine

OPS

TABLE OF CONTENTS — 13 EXECUTION MODULES

TABLE OF CONTENTS

◆◆◆

◆ Module 0 · Pre-Migration Audit and Inventory

◆ Module 1 · Sizing and Capacity Calculation

◆ Module 2 · Preparation and Pre-Flight

◆ Module 3 · Corosync Cluster and QDevice D3

◆ Module 4 · Ceph Stretched End-to-End

◆ Module 5 · SDN Network VXLAN/EVPN

◆ Module 6 · Security, Hardening and IAM

◆ Module 7 · HA, Fencing and Affinity

◆ Module 8 · v2v Migration Step by Step

◆ Module 9 · PBS Backup and Immutability

◆ Module 10 · Observability

◆ Module 11 · Failure Drills (Chaos Engineering)

◆ Module 12 · Continuous Operations and Acceptance

◆◆◆

◆ HOW TO USE THIS GUIDE

Each module follows the same structure: objective, execution commands, complete configuration files, and a validation checklist at the end of the section. Modules are designed to be followed in order — each subsequent module assumes the previous module's validations are complete. Command blocks are generic and must be adapted to your organisation's naming conventions, IP address ranges and security conventions before execution.

This document is the operational follow-up to the reference architecture study. It does not replace support from an integrator or architect on a real high-stakes migration project — it provides the reproducible execution skeleton.

OPS

MODULE 0 · PRE-MIGRATION AUDIT AND INVENTORY

PRE-MIGRATION AUDIT AND INVENTORY

◆

Before any action on Proxmox, a thorough inventory of the existing VMware environment is the number one condition for success. Most documented migration incidents in the industry stem from an unidentified application dependency upstream — not from a technical flaw in the target platform.

0.1 · VM Inventory via PowerCLI

Full export script of the vCenter inventory to CSV, including virtual hardware configuration, SPBM storage policies and guest tools status.

Connect-VIServer -Server vcenter.local
Get-VM | Select-Object Name, NumCpu, MemoryGB,
  @{N="Datastore";E={($_ | Get-Datastore).Name -join ","}},
  @{N="SPBMPolicy";E={(Get-SpbmEntityConfiguration -VM $_).StoragePolicy.Name}},
  @{N="ToolsStatus";E={$_.ExtensionData.Guest.ToolsStatus}},
  @{N="OS";E={$_.ExtensionData.Guest.GuestFullName}},
  PowerState | Export-Csv -Path vm_inventory.csv -NoTypeInformation

0.2 · Criticality Classification (Tier 1/2/3)

1Tier 1 (Critical) · RPO < 15 min, RTO < 1h. Transactional databases, ERP, payroll systems.

2Tier 2 (Important) · RPO < 1h, RTO < 4h. Non-critical business applications, intranet.

3Tier 3 (Non-critical) · RPO < 24h, RTO < 24h. Dev/test environments, sandboxes.

0.3 · NSX-T Network Dependency Mapping

Before replacing NSX-T with the Proxmox SDN, document the exact topology: logical segments, active DFW rules, T0/T1 gateways, anti-spoofing tables.

curl -k -u admin:password -X GET \
  "https://nsx-manager.local/policy/api/v1/infra/segments" \
  -H "Content-Type: application/json" > nsx_segments.json

curl -k -u admin:password -X GET \
  "https://nsx-manager.local/policy/api/v1/infra/domains/default/security-policies" \
  -H "Content-Type: application/json" > dfw_rules.json

0.4 · SPBM Storage Policy Audit

List the SPBM policies in use and their future mapping to Ceph CRUSH rules — this step determines the complexity of Module 4.

Get-SpbmStoragePolicy | Select-Object Name, AnyOfRuleSets, Description

0.5 · Windows Server Licensing Audit

For each Windows Server VM, check the licensing mode (per physical core, Datacenter, licence mobility) before migrating to KVM — a frequent and costly compliance gap if discovered after migration.

Complete CSV inventory generated and manually verified on 10% of VMs

Tier 1/2/3 classification validated with business teams

NSX-T export (segments + DFW) archived and documented

SPBM → CRUSH rules mapping table drafted

Windows Server licence compliance audit completed

OPS

MODULE 1 · SIZING AND CAPACITY CALCULATION

SIZING AND CAPACITY CALCULATION

◆

1.1 · Ceph Sizing Formula — Usable Capacity

Usable capacity depends on the chosen replication factor. For a stretched architecture with size=4 (2 copies per site) and min_size=2:

Usable_capacity = Total_raw_capacity / replication_factor
Usable_capacity = Total_raw_capacity / 4

Example: 24 OSDs of 4 TB each = 96 TB raw → Real usable capacity = 96 / 4 = 24 TB exploitable

1.2 · Required RAM per Ceph Node

Sizing rule: budget approximately 1 GB of RAM per TB of managed OSD, plus a 30% margin for recovery peaks.

Minimum_Ceph_RAM = (OSD_volume_per_node_in_TB × 1 GB) × 1.3

Example: a node with 4 OSDs of 4 TB = 16 TB per node → Minimum_Ceph_RAM = (16 × 1) × 1.3 = 20.8 GB dedicated to Ceph

1.3 · Compute Capacity Calculation (vCPU/vRAM)

Method for calculating the target consolidation ratio while accounting for the HA margin (loss of one site).

Available_vCPU_per_site = (Physical_cores × Nodes_per_site) × oversubscription_ratio
Required_HA_margin = Total_VM_load / Active_sites_after_loss

With 2 active sites and tolerance for the loss of 1 site: each site alone must be able to absorb 100% of the total load.

1.4 · Reusable TCO Calculator

Line item

Formula

Variable to fill in

Hardware CAPEX

Nodes × Unit_cost

Vendor quote

Annual vendor support

Nodes × Support_cost

~€1,400/year/node (rough order)

Double-run transition

VMware_cost × Transition_months

Variable per existing contract

Net annual saving

VMware_cost - Proxmox_support

Ceph usable capacity calculation documented and validated per site

Ceph RAM calculated and compared against ordered hardware specs

HA margin paper-tested: a single site absorbs 100% of the load

TCO table filled in with the project's actual figures

OPS

MODULE 2 · PREPARATION AND PRE-FLIGHT

PREPARATION AND PRE-FLIGHT

◆

2.1 · Proxmox VE Installation — Reproducible Deployment

Use the official ISO in graphical mode or via an answer file for automated, reproducible deployment across multiple nodes.

[global]
keyboard = "en-us"
country = "fr"
fqdn = "pve-d1-node1.lan.local"
timezone = "Europe/Paris"
root_password = "CHANGE_ME"

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid10"

2.2 · Network Configuration — Bonding and VLANs

Each node must have at least 2 network interfaces in active-active bonding (LACP) for Ceph back-end traffic, separated from the management network.

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer3+4

auto vmbr0
iface vmbr0 inet static
    address 10.10.1.11/24
    bridge-ports bond0
    bridge-stp off

2.3 · Inter-Site Network Benchmark — iperf3

Validate actual latency and bandwidth between D1 and D2. Stretched Ceph tolerates up to 10ms RTT.

# On D2 (server)
iperf3 -s -p 5201

# On D1 (client) — 60 seconds, 4 parallel streams
iperf3 -c 10.20.1.11 -p 5201 -t 60 -P 4

# Latency/jitter test
ping -c 100 10.20.1.11 | tail -5

Measured throughput ≥ 25 Gbps on the dedicated inter-site link

Average RTT latency < 10ms, stable jitter

2.4 · Raw Storage Benchmark — FIO

Raw performance test of NVMe disks before Ceph integration, to catch any underperforming disk before it becomes a cluster bottleneck.

fio --name=baseline_test --ioengine=libaio --rw=randwrite \
  --bs=4k --direct=1 --size=10G --numjobs=4 --iodepth=64 \
  --runtime=60 --group_reporting --filename=/dev/nvme0n1

Random 4k write IOPS consistent with manufacturer disk specs

Average write latency below 1ms on the disk alone

OPS

MODULE 2 · PREPARATION AND PRE-FLIGHT

PREPARATION AND PRE-FLIGHT

◆

2.5 · Hardware Pre-Flight Checklist

HBA controllers in IT/Pass-through mode (no active hardware RAID)

Power Loss Protection (PLP) verified on every SSD destined for Ceph

Hardware watchdog active and tested (/dev/watchdog accessible)

BIOS set to performance mode, C-States disabled on compute nodes

OPS

MODULE 3 · COROSYNC CLUSTER AND QDEVICE D3

COROSYNC CLUSTER AND QDEVICE D3

◆

3.1 · Forming the Proxmox Cluster

The first node initialises the cluster, subsequent nodes join it — creating the distributed configuration database (pmxcfs) shared across all nodes.

# On pve-d1-node1, initialisation
pvecm create cluster-stretched --link0 10.10.1.11

# On each subsequent node, joining
pvecm add 10.10.1.11 --link0 10.10.1.12

pvecm status shows the expected total number of nodes with quorum OK

3.2 · Deploying the QDevice on D3

The QDevice runs on an independent machine hosted at D3, fully separated from the D1-D2 network path.

# On the D3 machine
apt install corosync-qnetd -y

# On a Proxmox cluster node
apt install corosync-qdevice -y
pvecm qdevice setup 10.30.1.5 -f

pvecm status shows the QDevice as an active voting member

3.3 · Verifying Vote Distribution

With 3 nodes per site (D1 + D2) plus the QDevice, the total is 7 votes. Quorum is calculated automatically by Corosync on absolute majority.

pvecm status
# Expected output: Expected votes: 7, Total votes: 7, Quorum: 4

3.4 · Simulation Test — Loss of Inter-Site Link

Before going to production, simulate the D1-D2 link cut to validate the expected failover behaviour (the site with the QDevice obtains the majority).

# Network cut simulation on a test node
iptables -A INPUT -s 10.20.0.0/24 -j DROP
iptables -A OUTPUT -d 10.20.0.0/24 -j DROP

The priority site (with QDevice agreement) retains quorum

The isolated site triggers self-fencing (watchdog) without manual intervention

Restoring the link automatically restores the full cluster

3.5 · End-of-Module Checklist

7-vote cluster operational and stable for more than 24h

Network cut test performed and documented with timestamp

Post-incident reconnection procedure validated for the ops team

OPS

MODULE 4 · CEPH STRETCHED END-TO-END · 1/2

CEPH STRETCHED END-TO-END

◆

4.1 · Initial Deployment via cephadm

cephadm is the recommended deployment tool for recent Ceph versions. It runs in a container and orchestrates installation across all nodes.

# Bootstrap the first monitor
cephadm bootstrap --mon-ip 10.10.1.11 --cluster-network 10.10.2.0/24

# Add hosts to the Ceph cluster
ceph orch host add pve-d1-node2 10.10.1.12
ceph orch host add pve-d1-node3 10.10.1.13
ceph orch host add pve-d2-node1 10.20.1.11
ceph orch host add pve-d2-node2 10.20.1.12
ceph orch host add pve-d2-node3 10.20.1.13

4.2 · Building the Per-Datacenter CRUSH Hierarchy

Create logical buckets of type datacenter to encapsulate each site's hosts — a prerequisite for enabling stretch mode.

ceph osd crush add-bucket D1 datacenter
ceph osd crush add-bucket D2 datacenter
ceph osd crush move D1 root=default
ceph osd crush move D2 root=default
ceph osd crush move pve-d1-node1 datacenter=D1
ceph osd crush move pve-d1-node2 datacenter=D1
ceph osd crush move pve-d1-node3 datacenter=D1
ceph osd crush move pve-d2-node1 datacenter=D2
ceph osd crush move pve-d2-node2 datacenter=D2
ceph osd crush move pve-d2-node3 datacenter=D2

ceph osd tree shows the D1/D2 hierarchy correctly populated

4.3 · Defining the Stretched CRUSH Rule

The rule enforces 2 copies on D1 and 2 copies on D2, guaranteeing full data survival in the event of a single-site loss.

ceph osd crush rule create-replicated stretched_rule default datacenter
ceph osd crush rule dump stretched_rule

4.4 · Activating Native Stretch Mode

Native stretch mode explicitly binds the D3 arbiter monitor and activates automatic failover behaviour.

ceph mon set election_strategy connectivity
ceph mon enable_stretch_mode pve-d3-mon stretched_rule datacenter

ceph -s shows the cluster in active "stretched" mode without warnings

OPS

MODULE 4 · CEPH STRETCHED END-TO-END · 2/2

CEPH STRETCHED — RBD POOL AND FAILOVER TEST

◆

4.5 · Creating the RBD Pool for Proxmox

# Create the VM storage pool
ceph osd pool create vm-storage 128 128 replicated stretched_rule
ceph osd pool set vm-storage size 4
ceph osd pool set vm-storage min_size 2
rbd pool init vm-storage

# Integration into Proxmox
pvesm add rbd ceph-stretched --pool vm-storage \
  --monhost "10.10.1.11 10.20.1.11 10.30.1.5"

4.6 · Real Failover Test — Simulated Loss of Site D1

Controlled shutdown of Ceph services on D1 to validate failover behaviour under realistic conditions.

systemctl stop ceph-osd.target
systemctl stop ceph-mon.target

ceph -s from D2 shows HEALTH_WARN with the pool degraded but writable

VMs on D2 continue operating without write interruption

On D1 service restart, recovery resynchronisation starts automatically

4.7 · End-of-Module Checklist

vm-storage pool operational with size=4, min_size=2 verified

D1 cut test performed successfully, D2 VMs uninterrupted

Post-incident resynchronisation time measured and documented

⚠ SRE ATTENTION POINT — INTER-SITE NETWORK LATENCY

Synchronous Ceph replication means a write operation incurs the round-trip inter-site latency (RTT). A 2ms RTT adds a fixed transaction cost of roughly 2ms. Dedicated fibre links and enterprise-class NVMe disks are essential to avoid degrading perceived application performance.

OPS

MODULE 5 · SDN NETWORK VXLAN/EVPN

SDN NETWORK VXLAN/EVPN

◆

5.1 · Enabling the Proxmox SDN Module

apt install frr frr-pythontools -y
systemctl enable frr --now

5.2 · Configuring the EVPN Zone

An EVPN-type SDN zone encapsulates traffic in VXLAN with a BGP control plane for route distribution between D1, D2 and D3.

# /etc/pve/sdn/zones.cfg
evpn: zone-stretched
    controller bgp-controller
    vrf-vxlan 10000
    exitnodes pve-d1-node1,pve-d2-node1

# /etc/pve/sdn/controllers.cfg
bgp: bgp-controller
    asn 65000
    node pve-d1-node1
    peers 10.20.1.11,10.30.1.5

5.3 · Defining Application VNets

Each VNet corresponds to a logical segment equivalent to an NSX-T segment, with an anycast gateway distributed across all hypervisors.

# /etc/pve/sdn/vnets.cfg
vnet: vnet-web
    zone zone-stretched
    tag 100

vnet: vnet-db
    zone zone-stretched
    tag 200

5.4 · Distributed Firewall (DFW Equivalent)

# /etc/pve/firewall/cluster.fw
[group sec-web-servers]
IN SSH(ACCEPT) -log info
IN HTTP(ACCEPT)
IN HTTPS(ACCEPT)
OUT DB_PORT(ACCEPT) -dest +db-servers-ipset

[ipset web-servers-ipset]
10.100.10.0/24
[ipset db-servers-ipset]
10.100.20.0/24

5.5 · Anti-Spoofing

pvesh set /nodes/pve-d1-node1/qemu/101/firewall/options --ipfilter 1 --macfilter 1

5.6 · Inter-Site Connectivity Test

vtysh -c "show bgp l2vpn evpn summary"
vtysh -c "show evpn vni"

EVPN routes learned on all three exit nodes (D1, D2, D3)

A VM on vnet-web at D1 successfully pings a VM on vnet-web at D2

Unauthorised inter-VNet traffic is blocked by the firewall

5.7 · End-of-Module Checklist

EVPN zone stable, BGP established between D1/D2/D3

Microsegmentation rules tested both positive and negative

Anti-spoofing active and verified on all critical VMs

OPS

MODULE 6 · SECURITY, HARDENING AND IAM

SECURITY, HARDENING AND IAM

◆

6.1 · Hardening Cluster SSH Access

# /etc/ssh/sshd_config
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes
AllowUsers admin@10.0.0.0/8

6.2 · Enabling 2FA (TOTP) on the Web Interface

Proxmox VE natively supports TOTP for management interface access, independent of SSH authentication. Configuration via Datacenter > Permissions > Two Factor.

6.3 · LDAP/Active Directory Integration

Integrate the cluster with the existing corporate directory to centralise access management and traceability.

pveum realm add ad-company --type ad \
  --server1 ad.company.local \
  --domain company.local \
  --base-dn "DC=company,DC=local" \
  --bind-dn "CN=svc-proxmox,OU=Service,DC=company,DC=local"

6.4 · Managing Ceph and PBS Secrets

Ceph authentication keys (cephx) and PBS encryption keys must be stored outside the cluster itself, in a dedicated secrets vault.

ceph auth get client.admin -o /root/ceph-admin-key.backup
chmod 600 /root/ceph-admin-key.backup

6.5 · Permissions and Roles (RBAC) per Team

pveum role add VM-Operator -privs "VM.PowerMgmt,VM.Console,VM.Monitor"
pveum acl modify /vms/101 --roles VM-Operator --users ops-team@ad-company

6.6 · Log Auditing and Compliance

echo "*.* @@siem.company.local:514" >> /etc/rsyslog.conf
systemctl restart rsyslog

6.7 · End-of-Module Checklist

SSH password authentication disabled and keys deployed

2FA active for all privileged administrator accounts

AD/LDAP integration functional, tested with a non-admin account

Ceph and PBS keys backed up outside the cluster in a dedicated vault

RBAC applied: no application account holds the full Administrator role

OPS

MODULE 7 · HA, FENCING AND AFFINITY

HA, FENCING AND AFFINITY

◆

7.1 · Configuring HA Groups

An HA group defines the set of nodes eligible to host a VM on failover, with an optional priority per node.

ha-manager groupadd group-tier1 \
  --nodes "pve-d1-node1:2,pve-d1-node2:2,pve-d2-node1:1,pve-d2-node2:1"

7.2 · Adding a VM to HA Management

ha-manager add vm:101 --group group-tier1 --max_restart 3 --max_relocate 2

ha-manager status shows VM 101 as "started" with its group correctly assigned

7.3 · Anti-Affinity Rules

Prevent two redundant VMs (e.g. two domain controllers) from ending up on the same physical node after an automatic failover.

affinity_rules:
  - type: anti-affinity
    name: ad-controllers-isolation
    vms: [101, 102]
    action_on_violation: migrate

7.4 · Hardware Watchdog and Self-Fencing

ls -la /dev/watchdog
wdctl /dev/watchdog

The /dev/watchdog device is present and accessible by the Proxmox HA service

7.5 · Dynamic Load Balancing Script (DRS-like)

In the absence of a native dynamic scheduler equivalent to DRS, a scheduled script queries the Proxmox REST API to evaluate load and trigger live migrations.

import requests

def get_node_load(node):
    r = requests.get(f"https://cluster:8006/api2/json/nodes/{node}/status",
                      verify=False, headers={"Authorization": "PVEAPIToken=..."})
    return r.json()["data"]["cpu"]

nodes = ["pve-d1-node1", "pve-d1-node2", "pve-d2-node1", "pve-d2-node2"]
loads = {n: get_node_load(n) for n in nodes}
overloaded = max(loads, key=loads.get)
underloaded = min(loads, key=loads.get)
if loads[overloaded] - loads[underloaded] > 0.3:
    print(f"Migration recommended: {overloaded} → {underloaded}")

7.6 · Real HA Failover Test

echo c > /proc/sysrq-trigger
# Triggers an immediate kernel crash to simulate a sudden hardware failure

The VM automatically restarts on an HA group node within 120 seconds

Anti-affinity rules are respected after the failover

7.7 · End-of-Module Checklist

All HA groups created and Tier 1/2 VMs correctly assigned

Watchdog tested and confirmed functional on every node

Node crash test performed with measured and documented RTO

OPS

MODULE 8 · V2V MIGRATION STEP BY STEP

V2V MIGRATION STEP BY STEP

◆

8.1 · Preparing the vCenter Connection in Proxmox

The native Proxmox import wizard connects directly to vCenter to list VMs available for migration, with no manual conversion step required.

pvesh create /nodes/pve-d1-node1/import/vmware \
  --vcenter vcenter.company.local --username svc-migration@vsphere.local

8.2 · Migrating a Linux VM — Worked Example

pvesh create /nodes/pve-d1-node1/import/vmware/import \
  --vm-id 201 --source-vm "app-linux-01" \
  --storage vm-storage --bridge vnet-web

The VM boots under Proxmox with disks accessible via RBD

The IP address and application services respond normally after boot

8.3 · Post-Migration Optimisation — VirtIO Drivers on Linux

lsmod | grep virtio
# Should show virtio_net, virtio_blk or virtio_scsi depending on disk config

8.4 · Migrating a Windows Server VM — Worked Example

The Windows procedure requires injecting VirtIO drivers before the first boot under Proxmox, or risk a blue screen at boot time.

# Run on the source Windows VM, before final cutover
pnputil -i -a "E:\viostor\2k22\amd64\viostor.inf"
pnputil -i -a "E:\NetKVM\2k22\amd64\netkvm.inf"

pvesh create /nodes/pve-d1-node1/import/vmware/import \
  --vm-id 202 --source-vm "app-windows-01" \
  --storage vm-storage --bridge vnet-web --scsihw virtio-scsi-pci

Successful boot with no INACCESSIBLE_BOOT_DEVICE error

Windows Device Manager reports no missing drivers

8.5 · Configuring vTPM for BitLocker

qm set 202 --tpmstate0 vm-storage:1,version=v2.0

The Windows VM detects the TPM 2.0 module in Device Manager

8.6 · Windows Licence Compliance Declaration

Document the underlying physical infrastructure change in the company's Software Asset Management (SAM) portal — a condition of Microsoft contractual compliance.

8.7 · End-of-Module Checklist

Pilot Linux VM migrated and validated in production over a full cycle

Pilot Windows VM migrated, VirtIO injected, vTPM active if applicable

SAM declaration completed for all migrated Windows Server VMs

OPS

MODULE 9 · PBS BACKUP AND IMMUTABILITY

PBS BACKUP AND IMMUTABILITY

◆

9.1 · Installing Proxmox Backup Server

PBS is installed standalone, ideally on infrastructure physically distinct from the production cluster to ensure isolation in case of compromise.

wget http://download.proxmox.com/debian/pbs/proxmox-release-bookworm.gpg \
  -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
echo "deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription" \
  > /etc/apt/sources.list.d/pbs.list
apt update && apt install proxmox-backup-server -y

9.2 · Creating the Backup Datastore

proxmox-backup-manager datastore create pbs-backup-pool /mnt/backup/pbs-store

9.3 · Connecting Proxmox VE to the PBS Datastore

pvesm add pbs pbs-storage --server pbs.company.local \
  --datastore pbs-backup-pool --username backup@pbs --fingerprint "XX:XX:XX..."

9.4 · GFS Retention Policy

# /etc/proxmox-backup/datastore.cfg
datastore: pbs-backup-pool
    path /mnt/backup/pbs-store
    keep-last 14
    keep-daily 7
    keep-weekly 4
    keep-monthly 12
    keep-yearly 1
    verify-new true

9.5 · Enabling WORM Immutability

Time-based locking on critical backups to prevent any deletion or modification, even by a compromised administrator account.

proxmox-backup-client protected-set <SNAPSHOT_ID> true \
  --repository pbs.company.local:pbs-backup-pool

9.6 · Replication to D3 (Pull Sync)

proxmox-backup-manager remote create pbs-d1 --host pbs.company.local --auth-id sync@pbs
proxmox-backup-manager sync-job create job-d3-sync \
  --remote pbs-d1 --remote-store pbs-backup-pool \
  --store pbs-backup-pool-d3 --schedule "daily"

9.7 · Full Restore Test

qmrestore <BACKUP_ID> 301 --storage vm-storage

Full restore completed within the target RTO defined for the VM's Tier

Restored VM boots and data integrity is confirmed

9.8 · End-of-Module Checklist

GFS policy applied and verified on the main datastore

WORM immutability enabled on Tier 1 backups

D3 replication functional, tested with a complete sync job

Full restore test documented with measured time

OPS

MODULE 10 · OBSERVABILITY

OBSERVABILITY

◆

10.1 · Installing the Collection Stack

VictoriaMetrics or Prometheus collect the metrics natively exposed by Proxmox VE and Ceph, with no additional agent on the hosts.

# /etc/pve/status.cfg
influxdb: metrics-server
    server metrics.company.local
    port 8089
    mtu 1450

10.2 · Ceph Exporter

ceph mgr module enable prometheus
ceph config set mgr mgr/prometheus/server_addr 0.0.0.0
ceph config set mgr mgr/prometheus/server_port 9283

curl http://localhost:9283/metrics returns Ceph metrics in Prometheus format

10.3 · Configuring Scrape Targets

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'ceph'
    static_configs:
      - targets: ['pve-d1-node1:9283', 'pve-d2-node1:9283']
  - job_name: 'proxmox'
    static_configs:
      - targets: ['pve-d1-node1:9221', 'pve-d2-node1:9221']

10.4 · Reference SLI/SLO Indicators

Indicator

Target threshold

Remediation action

Inter-site WAN latency

< 10ms (p99)

P1 alert, suspend live migrations

Ceph write latency

< 15ms (p99)

Check NVMe disks and back-end network

SSD wearout

< 85%

P3 alert, schedule preventive replacement

Corosync quorum

100% available

Immediate P1 alert

10.5 · Prometheus Alert Rules

groups:
  - name: ceph_alerts
    rules:
      - alert: CephOSDWearoutWarning
        expr: ceph_disk_wearout_ratio > 0.85
        for: 2h
        labels: {severity: warning}
        annotations: {summary: "Critical OSD SSD wearout detected"}

10.6 · End-of-Module Checklist

Prometheus/VictoriaMetrics stack operational and collecting from all nodes

All SLI/SLO alert rules active and tested

Grafana dashboard validated with the ops team as a daily tool

OPS

MODULE 11 · FAILURE DRILLS (CHAOS ENGINEERING)

FAILURE DRILLS — LIGHTWEIGHT CHAOS ENGINEERING

◆

The scenarios below should be executed in pre-production and then repeated in production during planned maintenance windows, at a minimum of one full cycle every 6 months.

11.1 · Scenario A — Total Loss of Datacenter D1

for node in pve-d1-node1 pve-d1-node2 pve-d1-node3; do
  ssh $node "systemctl stop pve-cluster corosync ceph.target"
done

D2 retains quorum (3 votes + 1 QDevice = 4 votes out of 7)

Ceph switches to degraded single-site mode on D2 with no write loss

Tier 1 VMs from D1 automatically restart on D2 within 120 seconds

Automatic Ceph resynchronisation on return with no manual intervention

11.2 · Scenario B — Inter-Site Link Failure D1-D2

# Block traffic to the remote site only
iptables -A INPUT -s 10.20.0.0/24 -j DROP
iptables -A OUTPUT -d 10.20.0.0/24 -j DROP

The QDevice assigns its vote to the configured priority site (D1 obtains 4 votes)

Isolated D2 self-fences via watchdog without concurrent VM startup

No data corruption detected after the link is restored

11.3 · Scenario C — Single OSD Disk Failure

ceph osd out osd.5
ceph osd down osd.5

Ceph automatically triggers rebalancing to the remaining OSDs

No application service interruption during rebalancing

Prometheus alert triggered in line with Module 10 rules

11.4 · Scenario D — Full Recovery After Major Disaster

Combination of the previous scenarios: complete loss of D1 AND a failed automatic failover, to validate the last-resort manual recovery procedure.

Manual procedure executable in under 4 hours (Tier 1 RTO)

Ops team capable of executing the procedure without the architect who designed the cluster

◆ POST-DRILL REPORT

Each drill must produce a short report: actual failover duration, deviations from expected behaviour, identified corrective actions.

11.5 · End-of-Module Checklist

All 4 scenarios executed at least once in pre-production

Post-drill report written and corrective actions tracked

Recurring production drill calendar validated by operations management

OPS

MODULE 12 · CONTINUOUS OPERATIONS AND ACCEPTANCE

CONTINUOUS OPERATIONS AND ACCEPTANCE

◆

12.1 · Updating the Proxmox VE Cluster

Updates are performed node by node, never simultaneously, to maintain quorum at all times during the operation.

pve6to7 --full
# If OK:
apt update && apt dist-upgrade -y
# Verify the node rejoins the cluster before moving to the next one
pvecm status

Each node rejoins the cluster with quorum intact before the next node is treated

12.2 · Updating Ceph

Strict order: monitors (MON) first, managers (MGR) next, then OSDs one at a time with a health check between each step.

ceph orch upgrade start --image quay.io/ceph/ceph:v18

ceph -s shows HEALTH_OK before moving to the next upgrade step

12.3 · Replacing a Failed Node

# Clean removal
ceph osd out osd.{id}
ceph osd crush remove osd.{id}
ceph auth del osd.{id}
ceph osd rm osd.{id}
pvecm delnode pve-d1-node3-failed

# Integrating the replacement
pvecm add 10.10.1.11
ceph orch host add pve-d1-node3-replacement 10.10.1.14

The new node joins the cluster with quorum, OSDs reintegrated and rebalancing complete

12.4 · Final Acceptance Checklist per Migrated VM

1I/O performance comparable to or better than VMware (comparative FIO test)

2All application services functional after a cold restart

3PBS backup completed successfully at least once

4HA failover test performed on this VM without error

5Prometheus/Grafana monitoring active and alerts configured

6Technical documentation updated (CMDB, network diagram)

7Formal sign-off obtained from the business application owner

12.5 · Runbooks for Handover to Operations Teams

Standard structure: observed symptom · diagnostic checks · decision — automatic action vs manual intervention required · escalation procedure · reference architect contact.

12.6 · End-of-Module Checklist

Cluster upgrade procedure documented and tested in pre-production

Node replacement procedure tested at least once under real conditions

Acceptance grid applied to 100% of Tier 1 VMs before final cutover

Runbooks written for the 5 most probable incident scenarios