ๆฌๆๅฎๆด่จ้ไธๆฌกๅพๅฎๆฉ Proxmox VE ๆดๅฑ็บ ไธ็ฏ้ป้ซๅฏ็จๅข้(3-Node HA Cluster) ็ๅฏฆๆฐๆญท็จ,
ๅ
ๅซๆๆไฝ ๅฏ่ฝ้ๅฐ็ๅ:ไธปๆฉๆนๅ、DHCP ่ฝๅบๅฎ IP、็ถฒๅกๆฉๆฅๅคฑๆ、pmxcfs ้ๆญป、
ๆ่ญไธๅๆญฅ(SSL 596)、่
ฆ่ฃ(Split-brain)้ ้ฒ,ไปฅๅๆ็ต็ Failover ๅฏฆๆธฌ 130 ็งๅๆๆๅ。
๐บ️ ไธ、็ฐๅขๆฆ่ฆฝ่็ฎๆจๆถๆง
| ็ฏ้ป | IP ไฝๅ | ่ง่ฒ |
|---|---|---|
| pve-1 | 192.168.1.11 | ไธป้็ฎ็ฏ้ป(ๅๅ:pve) |
| pve-2 | 192.168.1.12 | ๅๆด็ฏ้ป |
| pve-3 | 192.168.1.13 | ๆ็ฅจ็ฏ้ป(1 CPU ไฝ่ณๆบ,ๅๅ:pve) |
| TrueNAS | 192.168.1.111 | NFS ๅฒๅญไผบๆๅจ(ISO & Backup) |
่จญ่จ็ฎๆจ:
- ไธๅฐ PVE ็ตๆๅข้,็ธฝ็ฅจๆธ 3 ็ฅจ
- pve-3 ไฝ็บ็ดๆ็ฅจๅก(Quorum Voter),ไธๆฟ่ผไปปไฝ VM
- VM 100(Ubuntu 22.04)ๅชๅจ pve-1 / pve-2 ไน้่ชๅๅๆ
- TrueNAS ๆไพ NFS ๅ ฑไบซๅฒๅญ,่ฎ VM ็กฌ็ข่ทจ็ฏ้ปๅฏ่ฆ
Quorum = ็ธฝ็ฅจๆธไธๅไปฅไธ。ๅ ฉ็ฏ้ปๆ,ไปปไธๅฐๆๆๅฉ 1/2 ็ฅจไธ่ถณ,ๆดๅๅข้้ๆญป。 ๅ ๅ ฅ pve-3 ๅพ,ๅณไฝฟ pve-2 ้ๆฉ,pve-1 + pve-3 ไปๆ 2/3 ็ฅจ,็ณป็ตฑ็นผ็บๆญฃๅธธ้ไฝ。
๐ง ไบ、ไธปๆฉๆนๅ่ๅบๅฎ IP
2.1 ็บไฝ่ฆๆนไธปๆฉๅ็จฑ?
Proxmox ๅข้็จไธปๆฉๅ็จฑๅ็ฏ้ป่ญๅฅ็ฌฆ。้ ่จญๅฎ่ฃๅพๅ็จฑ็บ pve,่ฅๅคๅฐๅๅๅฐฑๆ่ก็ช、ๆ่ญ้ฏ่ชค、SSH ไบไฟกๅคฑๆ。
2.2 ไธ้ตๆนๅ่ ณๆฌ(pve → pve-1)
# ไฟฎๆน hostname
sed -i 's/pve/pve-1/g' /etc/hostname
# ไฟฎๆน /etc/hosts(ๆ้้ต)
sed -i 's/pve/pve-1/g' /etc/hosts
# ้ๅๅพ้ท็งป VM ่จญๅฎ็ฎ้
reboot
# ้ๅๅพๅท่ก:
cp -a /etc/pve/nodes/pve/* /etc/pve/nodes/pve-1/
# ๆดๆฐ SSL ๆ่ญ
pvecm updatecerts -f
2.3 DHCP ่ฝๅบๅฎ IP(ไปฅ pve-1 ็บไพ)
DHCP ๆ่ฎ็ฏ้ปๆฏๆฌก้ๅ IP ๅฏ่ฝๆน่ฎ,ๅฐ่ด corosync.conf ็ด้็ IP ๅคฑๆ,ๅข้้่จๆท้。
cat <<EOF > /etc/network/interfaces
auto lo
iface lo inet loopback
iface nic0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.11/24
gateway 192.168.1.1
bridge-ports nic0
bridge-stp off
bridge-fd 0
EOF
systemctl restart networking && ifreload -a
ๆฅ่ๅๆญฅ /etc/hosts:
# pve-1
sed -i '/pve-1/d' /etc/hosts
echo '192.168.1.11 pve-1.proxmox.com pve-1' >> /etc/hosts
# pve-2
sed -i '/pve-2/d' /etc/hosts
echo '192.168.1.12 pve-2.proxmox.com pve-2' >> /etc/hosts
corosync.conf ่ฃก็่ IP ไน่ฆไธ่ตทๆดๆฐ,ไธฆๅฐ config_version ๅ 1,ๅฆๅๅข้่ชไธๅบ่ชๅทฑ。๐️ ไธ、็ตๅปบ 3-Node ๅข้
3.1 ๅปบ็ซๅข้(ๅจ pve-1 ๅท่ก)
pvecm create MyCluster
3.2 ๅ ถไป็ฏ้ปๅ ๅ ฅ
# ๅจ pve-2 / pve-3 ๅๅฅๅท่ก
pvecm add 192.168.1.11
# ่ผธๅ
ฅ yes ็ขบ่ช SSH ้ฃ็ท,ๅ่ผธๅ
ฅ pve-1 ็ root ๅฏ็ขผ
3.3 pve-3 ๅ ๅ ฅๅ็ๆธ ็(้ฟๅ ่ๅข้ๆฎ็)
systemctl stop pve-cluster corosync
killall -9 pmxcfs 2>/dev/null
rm -f /etc/pve/corosync.conf
rm -rf /etc/corosync/*
systemctl start pve-cluster
# ็ขบ่ชไนพๆทจๅพๅ ๅ
ฅ
pvecm add 192.168.1.11
3.4 ็ขบ่ชๅข้็ๆ
pvecm status
# ้ ๆ่ผธๅบ้้ป:
# Quorate: Yes
# Total votes: 3
# Nodes: 3
3.5 ๅฅไปถๅบซไฟฎๆญฃ(401 Unauthorized ่็)
rm -rf /etc/apt/sources.list.d/*
echo 'deb http://download.proxmox.com/debian/pve trixie pve-no-subscription' \
> /etc/apt/sources.list.d/pve-no-sub.list
echo 'deb http://download.proxmox.com/debian/ceph-squid trixie no-subscription' \
> /etc/apt/sources.list.d/ceph-no-sub.list
# ่ฃๅ Debian ๅบ็คๅฅไปถๅบซ
cat <<EOF > /etc/apt/sources.list
deb http://deb.debian.org/debian trixie main contrib non-free-remote-annex
deb http://deb.debian.org/debian trixie-updates main contrib non-free-remote-annex
deb http://security.debian.org/debian-security trixie-security main contrib
EOF
apt update && apt install vim -y
๐ ๅ、้็่ๅ้ก:ๅข้้ๆญป่ๆ่ญๅๆญฅ
้ๆฏๆดๅ้็จๆๆฃๆ็้จๅ。ไธๅ็็ๅพๅพๅๆๅบ็พ、ไบ็บๅ ๆ:
| ็็ | ๆ นๆฌๅๅ |
|---|---|
Write error (file system full?) | /etc/pve ่ขซ pmxcfs ้ๅฎ็บๅฏ่ฎ |
Cannot initialize CMAP service | corosync ๆๅๅฎๅ จๅดฉๆฝฐ,pvecm ๆไปค็กๆณไฝฟ็จ |
SSL certificate verify failed (596) | IP ่ฎๆดๅพ,่ๆ่ญ่ๆฐ IP ไธๅน้ |
no quorum! | ็ฏ้ปๅคฑ่ฏ็ฅจๆธไธ่ถณ,ๆ็ตไปปไฝๅฏซๅ ฅๆไฝ |
4.1 ่งฃ้ๆต็จ:pmxcfs ๆฌๅฐๆจกๅผ
# ๅๆญขๆๆ่ก็ชๆๅ
systemctl stop pve-ha-lrm pve-ha-crm corosync pve-cluster
killall -9 pmxcfs
# ไปฅๆฌๅฐๆจกๅผๆ่ผ(็น้ Quorum ๆชขๆฅ,ๆขๅพฉๅฏๅฏซ)
pmxcfs -l
# ไฟฎๆน corosync.conf(IP ่็ๆฌ่)
vim /etc/pve/corosync.conf
# ๆ :wq! ๅผทๅถๅฒๅญ
# ๆขๅพฉๆญฃๅธธๆจกๅผ
killall -9 pmxcfs
systemctl start pve-cluster corosync
pvecm updatecerts -f
pmxcfs -l ๆไฝ็จๅๅฐ。่ฅ้กฏ็คบ「unable to acquire pmxcfs lock」,ๅ
ๅท่ก killall -9 pmxcfs ๆธ
้คๆฎ็็จๅบๅ่ฉฆ。4.2 ไธ้ตๆๆด่ ณๆฌ
cat << 'EOF' > /tmp/pve_rescue.sh
#!/bin/bash
systemctl stop pve-ha-lrm pve-ha-crm corosync pve-cluster 2>/dev/null
killall -9 pmxcfs 2>/dev/null
pmxcfs -l
sleep 2
# ไฟฎๆญฃ corosync IP(ไพๅฏฆ้ๆ
ๆณไฟฎๆน)
sed -i 's/192.168.1.107/192.168.1.11/g' /etc/pve/corosync.conf
sed -i 's/192.168.1.130/192.168.1.12/g' /etc/pve/corosync.conf
sed -i 's/config_version: .*/config_version: 10/' /etc/pve/corosync.conf
# ไฟฎๆญฃ NFS ๅฒๅญ IP
sed -i 's/server .*/server 192.168.1.111/' /etc/pve/storage.cfg
# ๆธ
้ค HA ้ฏ่ชค้ๅฎ
rm -f /etc/pve/ha/resources.cfg
killall -9 pmxcfs
systemctl start pve-cluster
sleep 2
systemctl start corosync
pvecm updatecerts -f
EOF
chmod +x /tmp/pve_rescue.sh
/tmp/pve_rescue.sh
๐ ไบ、็ถฒ่ทฏๆฉๆฅ vmbr0 ๆ ้ๆ้ค
5.1 ็็
no physical interface on bridge 'vmbr0'
network script /usr/libexec/qemu-server/pve-bridge failed with status 6400
TASK ERROR: start failed: QEMU exited with code 1
ไปฃ่กจ vmbr0 ๆฒๆ็ถๅฎๅฏฆ้ซ็ถฒๅก,VM ็กๆณๅๅพ็ถฒ่ทฏๅบๅฃ。
5.2 ่จบๆท
ip link show
# ็ขบ่ช nic0(ๆ enp... )็ๆ
็บ UP,ไธ vmbr0 ๆ้กฏ็คบ master ้่ฏ
5.3 ๆฐธไน ไฟฎๅพฉ
cat <<NET_EOF > /etc/network/interfaces
auto lo
iface lo inet loopback
iface nic0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.1.11/24
gateway 192.168.1.1
bridge-ports nic0
bridge-stp off
bridge-fd 0
NET_EOF
ifreload -a
vmbr0 ๅฟ
้ ไฝฟ็จ็ธๅ็ๆฉๆฅ็ตๆง,ๅฆๅ VM ้ท็งปๅพๆๅคฑๅป็ถฒ่ทฏ้ฃ็ท。๐ฟ ๅ ญ、NFS ๅฒๅญๆดๅ(TrueNAS / OMV)
6.1 ๅจๅฒๅญไผบๆๅจ่จญๅฎ NFS Export
่ฅ showmount -e <IP> ้กฏ็คบ No file systems exported!,้ๆๅๅ ๅ
ฅๅฏๅบ่ฆๅ:
cp /etc/exports /etc/exports.bak
echo '/srv/dev-disk-by-uuid-.../ISO 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)' \
>> /etc/exports
exportfs -arv
showmount -e localhost
no_root_squash ๅฐ PVE ่ณ้้่ฆ:PVE ้ ่จญไปฅ root ๅญๅ NFS,่ฅ็ผบๅฐๆญค้ธ้
ๆๅ ๆฌ้่ขซๆ่็กๆณๅไปฝ。6.2 ๅจ PVE ๅ ๅ ฅ NFS ๅฒๅญ
Datacenter → Storage → Add → NFS,ๅกซๅ ฅไปฅไธ่ณ่จ:
| ๆฌไฝ | ๅกซๅ ฅๅผ |
|---|---|
| ID | ISO(ไปปๆๅ็จฑ) |
| Server | 192.168.1.111 |
| Export | ไธๆ้ธๅฎ่ชๅๅตๆธฌ่ทฏๅพ |
| Content | ISO image, VZDump backup, Disk image |
6.3 ่จบๆทๆไปค
showmount -e 192.168.1.111 # ๆธฌ่ฉฆ NFS ไผบๆๅจๆฏๅฆๆ้้
pvesm status # ๆ่ผ็ๆ
pvesm set ISO --disable 0 # ๅผทๅถๅ็จๅฒๅญ
mount -a # ้ๆฐๆ่ผ
⚙️ ไธ、HA ่จญๅฎ่ Node Affinity ่ฆๅ
7.1 HA ๆ ธๅฟๅ ไปถ
| ๅ ไปถ | ๅ่ฝ |
|---|---|
| CRM(ๅข้่ณๆบ็ฎก็ๅจ) | ่ทจ็ฏ้ปๅ่ชฟ,ๆฑบๅฎ VM ่ฆๅจๅชๅฐ่ท |
| LRM(ๆฌๅฐ่ณๆบ็ฎก็ๅจ) | ๅจๆฌๆฉๅท่ก้ๆฉ、้ๆฉ、้ท็งป |
| Master | ็ฑ CRM ๅๆ ้ธๅบ็ๆๆฎ็ฏ้ป |
7.2 ๅ ๅ ฅ HA ่ณๆบ
Datacenter → HA → Resources → Add,้ธๆ vm:100,่จญๅฎ Max. Restart ่ Max. Relocate。
7.3 ่จญๅฎ Node Affinity(ๆ้ค pve-3)
ๆฐ็ Proxmox ๅทฒๅฐ Groups ๆดๅ่ณ Rules。้้ Node Affinity ่ฆๅๆ้ค็นๅฎ็ฏ้ป:
Datacenter → HA → Affinity Rules → Add → Node Affinity
| ๆฌไฝ | ่จญๅฎ |
|---|---|
| HA Resources | vm:100 |
| Nodes | pve-1, pve-2(ไธ้ธ pve-3) |
7.4 ไฟฎๅพฉ LRM「Dead」็ๆ
่ฅ HA Status ้กฏ็คบ old timestamp - dead?,้ๅ HA ๆๅ:
systemctl restart pve-ha-lrm pve-ha-crm
๐ง ๅ ซ、่ ฆ่ฃ(Split-brain)้ ้ฒๆฉๅถ
8.1 ไป้บผๆฏ่ ฆ่ฃ?
ๅข้ๅ ็ถฒ่ทฏไธญๆทๅ่ฃๆๅ ฉ็ต,ๅ่ชไปฅ็บๅฐๆนๆญปๆ。่ฅๅ ฉ็ตๅๆๅฐๅไธไปฝ็กฌ็ขๅฏซๅ ฅ, ๆๅฐ่ด ๅด้็่ณๆๆๆฏ(Data Corruption)。
8.2 Proxmox ็้ฒ่ญทๆฉๅถ
- ๅชๆ็ฅจๆธ่ถ ้็ธฝ็ฅจๆธไธๅ็้ฃไธ็ตๆ่ฝ็นผ็บ้ไฝ
- ็ฅจๆธไธ่ถณ็้ฃไธ็ต่ชๅ้ฒๅ ฅๅฏ่ฎ้ๅฎ(็กๆณๅๅ VM)
- Watchdog ๆฉๅถ:LRM 10 ็งๅ งๆช้็ฝฎ็้็,็ณป็ตฑๅผทๅถ้ๅ็ฏ้ป
8.3 2-Node vs 3-Node ๅทฎ็ฐ
| ๆ ๅข | 2-Node ๅข้ | 3-Node ๅข้ |
|---|---|---|
| 1 ๅฐ้ๆฉ | ๅฉ 1/2 ็ฅจ → ❌ ้ๆญป | ๅฉ 2/3 ็ฅจ → ✅ ๆญฃๅธธ้ไฝ |
| ้ๆๅ่งฃ้ | ๆฏ(pvecm expected 1) | ๅฆ,่ชๅ็ถญๆ Quorum |
| ๅฎน้ฏ่ฝๅ | 0 ๅฐ(ไปปไฝๆ ้้ฝ้ๆญป) | 1 ๅฐ |
๐ฅ ไน、Failover ๅฏฆๆธฌ็ด้
9.1 ๅๅง็ๆ
| ้ ็ฎ | ็ๆ |
|---|---|
| VM 100 | pve-2 ไธ้่ก,Ubuntu 22.04 Server |
| pve-1 | active,ๅพ ๅฝไธญ |
| pve-2 | active,ๆฟ่ผ VM 100 |
| pve-3 | active,็ดๆ็ฅจ(Master ็ฑ pve-3 ๆไปป) |
| Quorum | OK(3/3 ็ฅจ) |
9.2 ๅๆๆๅบๅๆ
| ๆ้(็ง) | ไบไปถ |
|---|---|
| T+0 | pve-2 ้ปๆบ้้ |
| T+10 | pve-2 ็ Watchdog ้พๆ,็ณป็ตฑ็ขบ่ช็ฏ้ป็กๅๆ |
| T+60 | Master(pve-3)ๅคๅฎ pve-2 ้พๆ(dead?) |
| T+65 | pve-1 ๅๅพ VM 100 ็่ณๆบ้ๅฎ |
| T+75~140 | VM 100 ๅจ pve-1 ๅทๅๅๅฎๆ,ๆๅๆขๅพฉ |
9.3 ๅฏฆๆธฌ่งๅฏๅฐ็ ha-manager status ่ผธๅบ
# ้ๆฉๅพ็ด 60 ็ง
quorum OK
master pve-3 (active, Sun Mar 1 15:59:14 2026)
lrm pve-1 (active, Sun Mar 1 15:59:22 2026)
lrm pve-2 (old timestamp - dead?, Sun Mar 1 15:56:58 2026)
lrm pve-3 (idle, Sun Mar 1 15:59:24 2026)
service vm:100 (pve-1, starting)
# ็ด 140 ็งๅพ
service vm:100 (pve-1, started)
9.4 ๅฏฆๆธฌ็ต่ซ
| ๆๆจ | ๅฏฆๆธฌ็ตๆ |
|---|---|
| ๆ ้ๅตๆธฌๆ้ | ็ด 60 ็ง |
| VM ๅๅๆ้ | ็ด 60~80 ็ง(ๅซ Ubuntu ้ๆฉ) |
| ็ธฝๅๆๆ้ | ็ด 130~140 ็ง(็ด 2 ๅ 20 ็ง) |
| ่ณๆไธ่ดๆง | ๆญฃๅธธ(NFS ๅ ฑไบซๅฒๅญ็ก้่ค่ฃฝ) |
| Node Affinity ๆๆๆง | VM ๆช่ทๅฐ pve-3 ✅ |
9.5 Failback(ๆญธไฝ)ๆต็จ
- ๅฐ pve-2 ้ๆฐ้ๆฉ
- pve-2 ๅ ๅๅข้,ๆขๅพฉ็ถ ็
- ็ณป็ตฑ่ชๅๅท่ก Live Migration,ๅฐ VM 100 ๅพ pve-1 ๆฌๅ pve-2
- ้ท็งป้็จไธ้ๆฉ(Online Migration),ๆๅไธไธญๆท
๐ ๅ、ๅฎๆดๆไปค้ๆฅ่กจ
ๅข้็ฎก็
pvecm status # ๆฅ็ๅข้็ๆ
่ Quorum
pvecm expected 1 # ๅผทๅถไปฅ 1 ็ฅจ็ถญๆ Quorum(2-Node ็ทๆฅ่งฃ้)
pvecm updatecerts -f # ๅผทๅถๆดๆฐๅ
จๅข้ SSL ๆ่ญ
pvecm add 192.168.1.11 # ๅ ๅ
ฅๅข้
HA ็ฎก็
ha-manager status # ๆฅ็ HA ่ฉณ็ดฐ็ๆ
ha-manager set vm:100 --state disabled # ๅ็จ VM ็ HA ็ฎก็
ha-manager remove vm:100 # ็งป้ค VM ็ HA ็ฎก็
systemctl restart pve-ha-lrm pve-ha-crm # ้ๅ HA ๆๅ
VM ็ฎก็
qm start 100 # ๅๅ VM
qm unlock 100 # ่งฃ้ค VM ้ๅฎ
qm set 100 --memory 2048 # ่จญๅฎ่จๆถ้ซ(Ubuntu ่ณๅฐ้ 2048)
qm migrate 100 pve-2 --online # ็ทไธ็ฑ้ท็งป
็ถฒ่ทฏ็ฎก็
ip link show # ๆฅ็็ถฒๅก็ๆ
ip link set nic0 up # ๅ็จ็ถฒๅก
brctl addif vmbr0 nic0 # ่จๆๅ ๅ
ฅๆฉๆฅ
ifreload -a # ๅฅ็จ็ถฒ่ทฏ่จญๅฎ
ๆๅ้ๅๅ่
# ๅข้้่จ
systemctl restart corosync pve-cluster
# Web UI ็ฎก็ไป้ข
systemctl restart pveproxy pvedaemon pvestatd
# HA ๆๅ
systemctl restart pve-ha-lrm pve-ha-crm
ๅฆๆไฝ ๅจๅฏฆไฝ้็จ้ๅฐๅก้ป(็ๆฌ、็ฐๅข、้ฏ่ชค่จๆฏ),ๆญก่ฟ็่จ่จ่ซ:
- ไฝ ็็ฐๅข็ๆฌ(PVE / Kernel / ็ฏ้ปๆธ)
- ไฝ ๅไบๅชไบๆญฅ้ฉ
- ้ฏ่ชค่จๆฏๆ็พ่ฑก(้ไธ้้ต log)