Code Monkey home page Code Monkey logo

pimox7's Introduction

Pimox - Proxmox V7 for the Raspberry Pi

Pimox is a port of Proxmox to the Raspberry Pi allowing you to build a Proxmox cluster of Rapberry Pi's or even a hybrid cluster of Pis and x86 hardware.

Requirements

  • Raspberry Pi 4
  • Internet connection via ethernet

Install from "scratch", RPiOS64bit Interactive Automatic Installer

  1. Flash and startup the latest image from https://downloads.raspberrypi.org/raspios_arm64/ .
  2. sudo -s
  3. curl https://raw.githubusercontent.com/pimox/pimox7/master/RPiOS64-IA-Install.sh > RPiOS64-IA-Install.sh
  4. chmod +x RPiOS64-IA-Install.sh
  5. ./RPiOS64-IA-Install.sh
  6. Follow the prompts

Manual installation

Prechecks

  1. Pre-installed Debian Bullseye based 64-bit OS (not 32bit)
  2. In /etc/network/interfaces, give the Pi a static IP address. You cannot use dhcp.
  3. In /etc/network/interfaces, remove any IPv6 addresses.
  4. In /etc/hostname, make sure the Pi has a name.
  5. In /etc/hosts, make sure this hostname corresponds to the static IP you previous set.
  6. Make sure the kernel-headers are installed.

Installation

  1. echo "deb https://raw.githubusercontent.com/pimox/pimox7/master/ dev/" > /etc/apt/sources.list.d/pimox.list
  2. curl https://raw.githubusercontent.com/pimox/pimox7/master/KEY.gpg | apt-key add -
  3. apt update
  4. apt install proxmox-ve (use a local attatched console! Network connections will be lost/reset during installation progress)

Notes

  1. This repo just contains the precompiled debian packages. The original Proxmox sources can be found at https://git.proxmox.com
  2. The (very minimally) patched sources to rebuild this can be found at https://github.com/pimox

pimox7's People

Contributors

aanon4 avatar dhrinkino avatar learnlinuxtv avatar maccadoo avatar thracx avatar wings22actual avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pimox7's Issues

Update to the next PVE-7.1-10

Hey there,

long time, no see :)

Short question:

Im using the PVE 7.1-10 on my Intel Nuc as a x64 Maschine for VMs.
Now i setup a HA Cluster with the PiMox (7.1-8).

I have the problem, that the migration between both nodes fails, the support of Proxmox said that it is likely because of the different versions of the qemu-server and pve-qemu-kvm.

Version on the 7.1-10: qemu-server (7.1-4), pve-qemu-kvm (6.1.1-2)
Version on the 7.1-8: qemu-server (7.1-4rpve1), pve-qemu-kvm (6.0.0-4)

Also the support recommend to upgrade the kernel of the PiMox to the same as in my Intel Nuc:
proxmox-ve: 7.1-1 (running kernel: 5.15.19-2-pve)

On the Pi its still:
proxmox-ve: 7.1-1 (running kernel: 5.10.83-1-pve)

Is there any chance u could upgrade the packages and also provide the new 5.15. kernel?
Or at least tell us how to compile the new Kernel?

Kind regards,

Install script

Install script does not work on fresh install of the 64 bit Raspbian OS from Raspberry pi imager.

Reproduction steps:
Burn latest 64 bit Raspbian image from Raspberry pi imager.
Use instructions.

Error:
Package proxmox-ve does not install correctly because proxmox-manager is not configure correctly

Work around:
follow instructions about remove localhost and other extra entries...
https://forum.proxmox.com/threads/problem-with-installing-proxmox-ve-on-debian.44316/
apt install proxmox-ve
then run the script

TASK ERROR: start failed: QEMU exited with code 1

After the successful installation of Proxmox server on Raspberry Pi 4 B (4GB RAM), navigated to the web browser and pointed the host server with port 8006 and I was successful at the login as well.

But when I tried to create the VM I was hit by the following error.

kvm: -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101: Bus 'ide.1' not found TASK ERROR: start failed: QEMU exited with code 1

Help me to fix this issue.

Network vmbr0 won't start after Update/power outage

Hi,
i've got a strange problem.
Today i updated my pi:

the following components has been updated
raspberrypi-bootloader:arm64 (1:1.20211118-1, 1:1.20211118-3), firmware-brcm80211:arm64 (1:20210315-3+rpt3, 1:20210315-3+rpt4), firmware-atheros:arm64 (1:20210315-3+rpt3, 1:20210315-3+rpt4), libraspberrypi0:arm64 (1:2+git20211125~155417+14b90ff-1, 1:2+git20211125~155417+14b90ff-2), libraspberrypi-bin:arm64 (1:2+git20211125~155417+14b90ff-1, 1:2+git20211125~155417+14b90ff-2), libraspberrypi-dev:arm64 (1:2+git20211125~155417+14b90ff-1, 1:2+git20211125~155417+14b90ff-2), libpostproc55:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1), libraspberrypi-doc:arm64 (1:2+git20211125~155417+14b90ff-1, 1:2+git20211125~155417+14b90ff-2), libavcodec58:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1), firmware-realtek:arm64 (1:20210315-3+rpt3, 1:20210315-3+rpt4), firmware-libertas:arm64 (1:20210315-3+rpt3, 1:20210315-3+rpt4), libavutil56:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1), firmware-misc-nonfree:arm64 (1:20210315-3+rpt3, 1:20210315-3+rpt4), libswscale5:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1), proxmox-mini-journalreader:arm64 (1.3-1, 1.3-1rpve1), raspberrypi-kernel-headers:arm64 (1:1.20211118-1, 1:1.20211118-3), libswresample3:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1), libavformat58:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1), raspberrypi-kernel:arm64 (1:1.20211118-1, 1:1.20211118-3), linux-libc-dev:arm64 (1:1.20211118-1, 1:1.20211118-3), libavfilter7:arm64 (7:4.3.3-0+rpt1+deb11u1, 7:4.3.3-0+rpt2+deb11u1)

after that i called
apt autoremove
which removed "ssl-cert:arm64"

and cleared out some old configs:

dpkg -l | grep rc

and i remembered, that i removed the configurations for

apt purge ifupdown
apt purge ssl-cert
apt purge systemd-timesyncd

Later this day i got a poweroutage and realized, that the pi won't start. I plugged it to a monitor and keyboard and figured out, that the network inferfaces are down,
If i ran
systemctl restart networking

the vmbr0 came up and all is working fine....

The last Bootlog and versionlog is attached
version.log
boot.log

is there anything else, i can provide?

USB Passwthough not working

Hi, not sure if usb passthrough wont work by design, or if it's just a bug.

steps to reproduce:

  • Fresh PI Os 64 Bit upgraded to bullseye
  • fresh pimox7
  • create a vm, in my case alpine linux aarch64

qemu config:

agent: 1,fstrim_cloned_disks=1
bios: ovmf
boot: order=scsi1;scsi0;net0
cores: 4
cpu: host
efidisk0: SSD:vm-580-disk-1,size=64M
memory: 2048
name: test
net0: virtio=06:B7:8E:29:59:6A,bridge=vmbr0
numa: 0
ostype: l26
parent: clean
scsi0: SSD:vm-580-disk-0,discard=on,size=20G,ssd=1
scsi1: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=49442ca9-9897-408b-bdd7-f1a4890aeb77
sockets: 1
usb0: host=0451:16c8
usb1: host=0a12:0001

The VM boots just fine, but the usb devices are stuck on the pi, they dont get passed through.

USB Device Passthrough not working

I installed Pimox on raspios_lite_arm64-2021-11-08-Image using the installer-script,
created a new machine configured like the example in docs/VM-configuration.md
and installed debian 11.2 using the current arm64-net.iso.

Everything works fine.
It was even possible to setup a working homeassistant-supervised-installation in this VM!

I tried to use the ConBee II USB Stick as well as some other USB Devices to pass through to the guest os but it is not recognized. In syslog it only tells me a speed mismatch but is trying to connect anyway.
I had to uncheck "use USB 3" otherwise it tries to connect to a pci device

Bildschirmfoto 2022-01-12 um 17 43 41
Bildschirmfoto 2022-01-12 um 17 44 45
.

Duplicate ID 'ehci' for device

Thanks a lot. It works like a charm. However I tried to run Home assistant OS as a VM, I got the following error :

kvm: -device usb-ehci,id=ehci,bus=pci.0,addr=0x1: Duplicate ID 'ehci' for device TASK ERROR: start failed: QEMU exited with code 1

My pi is powered by poe hat and there is only one USB connected for the SSD where Proxmox is installed. No SD card in there.

Would you have any idea how I can solve this issue?

Display output is not active

I sucessfully installed pimox on debian11 and tried to install debian 10 for Home assistant.
At first, debian 10 install screen was appered and I selected install
But right after, I only can see black screen with below comment.
'Display output is not active'
I tried serverl display option but only default option is work.
The others option didn't show even the first install screen.
What should I do?

Successfull upgrade from Pimox6to7

I set it as issue :) but it is a success Message.
I have edit all source files to Bullseye and Pimox7 and made a dist-upgrade.
All went smooth and working fine now with v7.
Big Thanks for your great work to contribute this Packages to us.

Raspberry CM4 support

Does this installation support raspberry Compute Module 4?

from the processor/ram standpoint it is equivalent to a Raspberry Pi4.

Support aarch64 (arm64 synonym)

Hi there! Thanks for this script. You've done a lot of groundwork that I've needed.

I have an ODroid N2+ (arm64) and I figured I could run the script on it, however the arch is aarch64. How could I rebuild the sources from all the repos? Do I need to clone all repos first, rebuild each project one-by-one then run sync?

Once I get it working, perhaps we can make the hard coded arm64 as uname -m.

Thanks!

Home Assistant OS VM not booting

I’m trying to create a Home Assistant OS virtual machine on PimoxV7 running on my RPi 4 with 4GB of RAM (I don’t know if it’s even possibile what I’m trying to do)

At the moment I tried 2 different attempts:

Neither method allows me to boot the image, ending on the UEFI shell. I even tried to change the boot order as described by this guide without success.

I leave some screenshots of the vm’s hardware and configuration:
Schermata 2022-03-28 alle 11 56 59
Schermata 2022-03-28 alle 11 57 10
Schermata 2022-03-28 alle 11 57 40
Schermata 2022-03-28 alle 11 58 03
Schermata 2022-03-28 alle 12 12 28

Could someone help me? Even just letting me know if it’s theoretically possible?

Firewall not applying rules

Hello,
First, thank you for your awesome work.

Today I noticed that the integrated firewall does not apply any changes and doesn't filter traffic. It looks like some module for the network bridge is missing or not loaded.

Output of pve-firewall status:

Status: enabled/running (pending changes)

pve-firewall compile:

create PVEFW-0-management-v4 (PWD31JmBcC/0aYW2jK2JsOwXDO8)
	create PVEFW-0-management-v4 hash:net family inet hashsize 64 maxelem 64 bucketsize 12
	add PVEFW-0-management-v4 x.x.x.0/24
create PVEFW-0-management-v6 (6g+lzHFoCegXcweHRfBY4vRsbOc)
	create PVEFW-0-management-v6 hash:net family inet6 hashsize 64 maxelem 64 bucketsize 12

iptables cmdlist:
create GROUP-default-IN (CNd1DuhyCdbmOVpU1vbYSJjWHqo)
	-A GROUP-default-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-default-OUT (tsnPkIAHd6d4iCYjG485zXP3eI0)
	-A GROUP-default-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A GROUP-default-OUT -d x.x.x.4 -p udp --dport 53 -g PVEFW-SET-ACCEPT-MARK
	-A GROUP-default-OUT -d x.x.x.4 -p tcp --dport 53 -g PVEFW-SET-ACCEPT-MARK
	-A GROUP-default-OUT -d x.x.x.0/8 -j DROP
	-A GROUP-default-OUT -d x.x.x.0/12 -j DROP
	-A GROUP-default-OUT -d x.x.x.0/16 -j DROP
create PVEFW-Drop (83WlR/a4wLbmURFqMQT3uJSgIG8)
	-A PVEFW-Drop  -j PVEFW-DropBroadcast
	-A PVEFW-Drop -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
	-A PVEFW-Drop -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
	-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
	-A PVEFW-Drop -p udp --dport 137:139 -j DROP
	-A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
	-A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
	-A PVEFW-Drop -p udp --dport 1900 -j DROP
	-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
	-A PVEFW-Drop -p udp --sport 53 -j DROP
create PVEFW-DropBroadcast (NyjHNAtFbkH7WGLamPpdVnxHy4w)
	-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
	-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
	-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
	-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
create PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
	-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
	-A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
	-A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-IN (BSfT3ROZgjc9bnx2RjYgT2ZBhfk)
	-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
	-A PVEFW-FWBR-IN -p tcp -j PVEFW-tcpflags
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out tap800i0 -j tap800i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out tap801i0 -j tap801i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out tap802i0 -j tap802i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out veth100i0 -j veth100i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out veth700i0 -j veth700i0-IN
create PVEFW-FWBR-OUT (a3XLOO96rl7sA8sks3wMKsfYY+8)
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in tap800i0 -j tap800i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in tap801i0 -j tap801i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in tap802i0 -j tap802i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in veth100i0 -j veth100i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in veth700i0 -j veth700i0-OUT
create PVEFW-HOST-IN (nYiug+DLaBrUddfKb/HgFQb82WM)
	-A PVEFW-HOST-IN -i lo -j ACCEPT
	-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
	-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
	-A PVEFW-HOST-IN -p tcp -j PVEFW-tcpflags
	-A PVEFW-HOST-IN -p igmp -j RETURN
	-A PVEFW-HOST-IN -i vmbr0 -j GROUP-default-IN
	-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A PVEFW-HOST-IN -i vmbro -s x.x.x.0/24 -p tcp --dport 8006 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 8006 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 5900:5999 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 3128 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 22 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v4 src -p tcp --dport 60000:60050 -j RETURN
	-A PVEFW-HOST-IN -j PVEFW-Drop
	-A PVEFW-HOST-IN -j DROP
create PVEFW-HOST-OUT (mu4KE7UJdZaYxpkjY5nKmIYfizc)
	-A PVEFW-HOST-OUT -o lo -j ACCEPT
	-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
	-A PVEFW-HOST-OUT -p igmp -j RETURN
	-A PVEFW-HOST-OUT -o vmbr0 -j GROUP-default-OUT
	-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A PVEFW-HOST-OUT -o vmbr0 -p tcp --dport 80 -j RETURN
	-A PVEFW-HOST-OUT -o vmbr0 -p tcp -j RETURN
	-A PVEFW-HOST-OUT -d x.x.x.0/24 -p tcp --dport 8006 -j RETURN
	-A PVEFW-HOST-OUT -d x.x.x.0/24 -p tcp --dport 22 -j RETURN
	-A PVEFW-HOST-OUT -d x.x.x.0/24 -p tcp --dport 5900:5999 -j RETURN
	-A PVEFW-HOST-OUT -d x.x.x.0/24 -p tcp --dport 3128 -j RETURN
	-A PVEFW-HOST-OUT -j PVEFW-Drop
	-A PVEFW-HOST-OUT -j DROP
create PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
	-A PVEFW-INPUT -j PVEFW-HOST-IN
create PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
	-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
create PVEFW-Reject (h3DyALVslgH5hutETfixGP08w7c)
	-A PVEFW-Reject  -j PVEFW-DropBroadcast
	-A PVEFW-Reject -p icmp -m icmp --icmp-type fragmentation-needed -j ACCEPT
	-A PVEFW-Reject -p icmp -m icmp --icmp-type time-exceeded -j ACCEPT
	-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
	-A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
	-A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
	-A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
	-A PVEFW-Reject -p udp --dport 1900 -j DROP
	-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
	-A PVEFW-Reject -p udp --sport 53 -j DROP
create PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
	-A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
create PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
	-A PVEFW-logflags  -j DROP
create PVEFW-reject (Jlkrtle1mDdtxDeI9QaDSL++Npc)
	-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
	-A PVEFW-reject -s 224.0.0.0/4 -j DROP
	-A PVEFW-reject -p icmp -j DROP
	-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
	-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
	-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
	-A PVEFW-reject  -j REJECT --reject-with icmp-host-prohibited
create PVEFW-smurflog (2gfT1VMkfr0JL6OccRXTGXo+1qk)
	-A PVEFW-smurflog  -j DROP
create PVEFW-smurfs (HssVe5QCBXd5mc9kC88749+7fag)
	-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
	-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
	-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
create PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
create tap800i0-IN (0X1WBDahrSg5QUIbxusNMrB/agQ)
	-A tap800i0-IN -p udp --sport 67 --dport 68 -j ACCEPT
	-A tap800i0-IN -j GROUP-default-IN
	-A tap800i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A tap800i0-IN -s x.x.x.0/24 -p tcp --dport 22 -j ACCEPT
	-A tap800i0-IN -j PVEFW-Drop
	-A tap800i0-IN -j DROP
create tap800i0-OUT (cc92hk+NKX0z9Y5RHqo/5ry4j8o)
	-A tap800i0-OUT -p udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A tap800i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A tap800i0-OUT -j GROUP-default-OUT
	-A tap800i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A tap800i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -j PVEFW-Drop
	-A tap800i0-OUT -j DROP
create tap801i0-IN (K359Au2B83LeGAXuozldyWma8IY)
	-A tap801i0-IN -p udp --sport 67 --dport 68 -j ACCEPT
	-A tap801i0-IN -j GROUP-default-IN
	-A tap801i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A tap801i0-IN -s x.x.x.0/24 -p tcp --dport 22 -j ACCEPT
	-A tap801i0-IN -j PVEFW-Drop
	-A tap801i0-IN -j DROP
create tap801i0-OUT (p2Z8+uCAp0o78gCqxPYjQn548pU)
	-A tap801i0-OUT -p udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A tap801i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A tap801i0-OUT -j GROUP-default-OUT
	-A tap801i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A tap801i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -p tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -j PVEFW-Drop
	-A tap801i0-OUT -j DROP
create tap802i0-IN (IM5H1EbEixrkfhgRAzRMT4tS7hQ)
	-A tap802i0-IN -p udp --sport 67 --dport 68 -j ACCEPT
	-A tap802i0-IN -j GROUP-default-IN
	-A tap802i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A tap802i0-IN -s x.x.x.0/24 -p tcp --dport 22 -j ACCEPT
	-A tap802i0-IN -j PVEFW-Drop
	-A tap802i0-IN -j DROP
create tap802i0-OUT (DDEHCCe73Rc12m6nIFyXFVnlNgM)
	-A tap802i0-OUT -p udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A tap802i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A tap802i0-OUT -j GROUP-default-OUT
	-A tap802i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A tap802i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -p tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -j PVEFW-Drop
	-A tap802i0-OUT -j DROP
create veth100i0-IN (bJyUpWfa1Gy8Uf8cpRIYNxhI8fw)
	-A veth100i0-IN -p udp --sport 67 --dport 68 -j ACCEPT
	-A veth100i0-IN -j GROUP-default-IN
	-A veth100i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A veth100i0-IN -p udp --dport 53 -j ACCEPT
	-A veth100i0-IN -p tcp --dport 53 -j ACCEPT
	-A veth100i0-IN -s x.x.x.0/24 -p tcp --dport 80 -j ACCEPT
	-A veth100i0-IN -j PVEFW-Drop
	-A veth100i0-IN -j DROP
create veth100i0-OUT (qOM8WmeirxF1gGG1DbxIEmY2Bes)
	-A veth100i0-OUT -p udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A veth100i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A veth100i0-OUT -j GROUP-default-OUT
	-A veth100i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A veth100i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -p tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -j PVEFW-Drop
	-A veth100i0-OUT -j DROP
create veth700i0-IN (95DDwIXA9ixmdBnPXTyRyntok8k)
	-A veth700i0-IN -p udp --sport 67 --dport 68 -j ACCEPT
	-A veth700i0-IN -j GROUP-default-IN
	-A veth700i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A veth700i0-IN -s x.x.x.0/24 -p tcp --dport 22 -j ACCEPT
	-A veth700i0-IN -j PVEFW-Drop
	-A veth700i0-IN -j DROP
create veth700i0-OUT (Ko4gSzqYb3rfjIsv+gGx06BLIXg)
	-A veth700i0-OUT -p udp --sport 68 --dport 67 -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A veth700i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A veth700i0-OUT -j GROUP-default-OUT
	-A veth700i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A veth700i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -j PVEFW-Drop
	-A veth700i0-OUT -j DROP

ip6tables cmdlist:
create GROUP-default-IN (CNd1DuhyCdbmOVpU1vbYSJjWHqo)
	-A GROUP-default-IN -j MARK --set-mark 0x00000000/0x80000000
create GROUP-default-OUT (m40rxGxRolSs4B26P5z+oKHDMIc)
	-A GROUP-default-OUT -j MARK --set-mark 0x00000000/0x80000000
create PVEFW-Drop (Jb79Uw7z1vZglIcV7QXA5uY/nbk)
	-A PVEFW-Drop -p tcp --dport 43 -j PVEFW-reject
	-A PVEFW-Drop  -j PVEFW-DropBroadcast
	-A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
	-A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
	-A PVEFW-Drop -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
	-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-Drop -p udp --match multiport --dports 135,445 -j DROP
	-A PVEFW-Drop -p udp --dport 137:139 -j DROP
	-A PVEFW-Drop -p udp --sport 137 --dport 1024:65535 -j DROP
	-A PVEFW-Drop -p tcp --match multiport --dports 135,139,445 -j DROP
	-A PVEFW-Drop -p udp --dport 1900 -j DROP
	-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
	-A PVEFW-Drop -p udp --sport 53 -j DROP
create PVEFW-DropBroadcast (8Krk5Nh8pDZOOc7BQAbM6PlyFSU)
	-A PVEFW-DropBroadcast -d ff00::/8 -j DROP
create PVEFW-FORWARD (qnNexOcGa+y+jebd4dAUqFSp5nw)
	-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
	-A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-in fwln+ -j PVEFW-FWBR-IN
	-A PVEFW-FORWARD -m physdev --physdev-is-bridged --physdev-out fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-IN (jEvMpjKi+QBNFQ7F7fa408lM/1E)
	-A PVEFW-FWBR-IN -p tcp -j PVEFW-tcpflags
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out tap800i0 -j tap800i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out tap801i0 -j tap801i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out tap802i0 -j tap802i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out veth100i0 -j veth100i0-IN
	-A PVEFW-FWBR-IN -m physdev --physdev-is-bridged --physdev-out veth700i0 -j veth700i0-IN
create PVEFW-FWBR-OUT (a3XLOO96rl7sA8sks3wMKsfYY+8)
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in tap800i0 -j tap800i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in tap801i0 -j tap801i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in tap802i0 -j tap802i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in veth100i0 -j veth100i0-OUT
	-A PVEFW-FWBR-OUT -m physdev --physdev-is-bridged --physdev-in veth700i0 -j veth700i0-OUT
create PVEFW-HOST-IN (wYO4RxtGWIi0jV12dP6ai3R27Js)
	-A PVEFW-HOST-IN -i lo -j ACCEPT
	-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
	-A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-solicitation -j RETURN
	-A PVEFW-HOST-IN -p icmpv6 --icmpv6-type router-advertisement -j RETURN
	-A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
	-A PVEFW-HOST-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
	-A PVEFW-HOST-IN -p tcp -j PVEFW-tcpflags
	-A PVEFW-HOST-IN -p igmp -j RETURN
	-A PVEFW-HOST-IN -i vmbr0 -j GROUP-default-IN
	-A PVEFW-HOST-IN -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 8006 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 5900:5999 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 3128 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 22 -j RETURN
	-A PVEFW-HOST-IN -m set --match-set PVEFW-0-management-v6 src -p tcp --dport 60000:60050 -j RETURN
	-A PVEFW-HOST-IN -j PVEFW-Drop
	-A PVEFW-HOST-IN -j DROP
create PVEFW-HOST-OUT (xDjizyEwaEN268ucLgIqQIUQ7vg)
	-A PVEFW-HOST-OUT -o lo -j ACCEPT
	-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
	-A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type router-solicitation -j RETURN
	-A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -j RETURN
	-A PVEFW-HOST-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -j RETURN
	-A PVEFW-HOST-OUT -p igmp -j RETURN
	-A PVEFW-HOST-OUT -o vmbr0 -j GROUP-default-OUT
	-A PVEFW-HOST-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A PVEFW-HOST-OUT -o vmbr0 -p tcp --dport 80 -j RETURN
	-A PVEFW-HOST-OUT -o vmbr0 -p tcp -j RETURN
	-A PVEFW-HOST-OUT -j PVEFW-Drop
	-A PVEFW-HOST-OUT -j DROP
create PVEFW-INPUT (+5iMmLaxKXynOB/+5xibfx7WhFk)
	-A PVEFW-INPUT -j PVEFW-HOST-IN
create PVEFW-OUTPUT (LjHoZeSSiWAG3+2ZAyL/xuEehd0)
	-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
create PVEFW-Reject (aL1nrxJk/u3XmTb3Am2eaM/3yCM)
	-A PVEFW-Reject -p tcp --dport 43 -j PVEFW-reject
	-A PVEFW-Reject  -j PVEFW-DropBroadcast
	-A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
	-A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type time-exceeded -j ACCEPT
	-A PVEFW-Reject -p icmpv6 -m icmpv6 --icmpv6-type packet-too-big -j ACCEPT
	-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
	-A PVEFW-Reject -p udp --match multiport --dports 135,445 -j PVEFW-reject
	-A PVEFW-Reject -p udp --dport 137:139 -j PVEFW-reject
	-A PVEFW-Reject -p udp --sport 137 --dport 1024:65535 -j PVEFW-reject
	-A PVEFW-Reject -p tcp --match multiport --dports 135,139,445 -j PVEFW-reject
	-A PVEFW-Reject -p udp --dport 1900 -j DROP
	-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
	-A PVEFW-Reject -p udp --sport 53 -j DROP
create PVEFW-SET-ACCEPT-MARK (Hg/OIgIwJChBUcWU8Xnjhdd2jUY)
	-A PVEFW-SET-ACCEPT-MARK  -j MARK --set-mark 0x80000000/0x80000000
create PVEFW-logflags (MN4PH1oPZeABMuWr64RrygPfW7A)
	-A PVEFW-logflags  -j DROP
create PVEFW-reject (etEECUYcgUdzuuO+LDP83pu0S8Y)
	-A PVEFW-reject -p icmpv6 -j DROP
	-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
	-A PVEFW-reject -p udp -j REJECT --reject-with icmp6-port-unreachable
	-A PVEFW-reject  -j REJECT --reject-with icmp6-adm-prohibited
create PVEFW-tcpflags (CMFojwNPqllyqD67NeI5m+bP5mo)
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
	-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
create tap800i0-IN (lTLjVGHVuubG8Qd8RtshZFxpHbo)
	-A tap800i0-IN -p udp --sport 547 --dport 546 -j ACCEPT
	-A tap800i0-IN -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
	-A tap800i0-IN -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
	-A tap800i0-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
	-A tap800i0-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
	-A tap800i0-IN -j GROUP-default-IN
	-A tap800i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A tap800i0-IN -j PVEFW-Drop
	-A tap800i0-IN -j DROP
create tap800i0-OUT (d2JztIS+moByZ00wpYT6fbxUTf4)
	-A tap800i0-OUT -p udp --sport 546 --dport 547 -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A tap800i0-OUT -p icmpv6 --icmpv6-type router-advertisement -j DROP
	-A tap800i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A tap800i0-OUT -p icmpv6 --icmpv6-type router-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -j GROUP-default-OUT
	-A tap800i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A tap800i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A tap800i0-OUT -j PVEFW-Drop
	-A tap800i0-OUT -j DROP
create tap801i0-IN (vnk8h8ss7S4w+VtyKAN2JdiaFEk)
	-A tap801i0-IN -p udp --sport 547 --dport 546 -j ACCEPT
	-A tap801i0-IN -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
	-A tap801i0-IN -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
	-A tap801i0-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
	-A tap801i0-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
	-A tap801i0-IN -j GROUP-default-IN
	-A tap801i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A tap801i0-IN -j PVEFW-Drop
	-A tap801i0-IN -j DROP
create tap801i0-OUT (6fFptpL+B7DbCrFRgToS1Fp4zOE)
	-A tap801i0-OUT -p udp --sport 546 --dport 547 -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A tap801i0-OUT -p icmpv6 --icmpv6-type router-advertisement -j DROP
	-A tap801i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A tap801i0-OUT -p icmpv6 --icmpv6-type router-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -j GROUP-default-OUT
	-A tap801i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A tap801i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -p tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
	-A tap801i0-OUT -j PVEFW-Drop
	-A tap801i0-OUT -j DROP
create tap802i0-IN (lMzGJ0+SMdqtoStyXvH1IPcbDCg)
	-A tap802i0-IN -p udp --sport 547 --dport 546 -j ACCEPT
	-A tap802i0-IN -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
	-A tap802i0-IN -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
	-A tap802i0-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
	-A tap802i0-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
	-A tap802i0-IN -j GROUP-default-IN
	-A tap802i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A tap802i0-IN -j PVEFW-Drop
	-A tap802i0-IN -j DROP
create tap802i0-OUT (OCNJfG5t8FjpmGXGo0UmTE96jg4)
	-A tap802i0-OUT -p udp --sport 546 --dport 547 -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A tap802i0-OUT -p icmpv6 --icmpv6-type router-advertisement -j DROP
	-A tap802i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A tap802i0-OUT -p icmpv6 --icmpv6-type router-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -j GROUP-default-OUT
	-A tap802i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A tap802i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -p tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
	-A tap802i0-OUT -j PVEFW-Drop
	-A tap802i0-OUT -j DROP
create veth100i0-IN (qK/wAlqYPsacG+UwfOSLuroaSZw)
	-A veth100i0-IN -p udp --sport 547 --dport 546 -j ACCEPT
	-A veth100i0-IN -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
	-A veth100i0-IN -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
	-A veth100i0-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
	-A veth100i0-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
	-A veth100i0-IN -j GROUP-default-IN
	-A veth100i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A veth100i0-IN -p udp --dport 53 -j ACCEPT
	-A veth100i0-IN -p tcp --dport 53 -j ACCEPT
	-A veth100i0-IN -j PVEFW-Drop
	-A veth100i0-IN -j DROP
create veth100i0-OUT (SEeXJUJonSWXjbLzlDSklhk8aAU)
	-A veth100i0-OUT -p udp --sport 546 --dport 547 -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A veth100i0-OUT -p icmpv6 --icmpv6-type router-advertisement -j DROP
	-A veth100i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A veth100i0-OUT -p icmpv6 --icmpv6-type router-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -j GROUP-default-OUT
	-A veth100i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A veth100i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -p tcp --dport 443 -g PVEFW-SET-ACCEPT-MARK
	-A veth100i0-OUT -j PVEFW-Drop
	-A veth100i0-OUT -j DROP
create veth700i0-IN (IO0SD3iWECj+s+cUILk9a5dptac)
	-A veth700i0-IN -p udp --sport 547 --dport 546 -j ACCEPT
	-A veth700i0-IN -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT
	-A veth700i0-IN -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT
	-A veth700i0-IN -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT
	-A veth700i0-IN -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT
	-A veth700i0-IN -j GROUP-default-IN
	-A veth700i0-IN -m mark --mark 0x80000000/0x80000000 -j ACCEPT
	-A veth700i0-IN -j PVEFW-Drop
	-A veth700i0-IN -j DROP
create veth700i0-OUT (G6pWIdizFZTxn0ASNUI6RF1nJtY)
	-A veth700i0-OUT -p udp --sport 546 --dport 547 -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -m mac ! --mac-source XX:XX:XX:XX:XX:XX -j DROP
	-A veth700i0-OUT -p icmpv6 --icmpv6-type router-advertisement -j DROP
	-A veth700i0-OUT -j MARK --set-mark 0x00000000/0x80000000
	-A veth700i0-OUT -p icmpv6 --icmpv6-type router-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -p icmpv6 --icmpv6-type neighbor-solicitation -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -p icmpv6 --icmpv6-type neighbor-advertisement -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -j GROUP-default-OUT
	-A veth700i0-OUT -m mark --mark 0x80000000/0x80000000 -j RETURN
	-A veth700i0-OUT -p tcp --dport 80 -g PVEFW-SET-ACCEPT-MARK
	-A veth700i0-OUT -j PVEFW-Drop
	-A veth700i0-OUT -j DROP

ebtables cmdlist:
create PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
	-A PVEFW-FORWARD -p IPv4 -j ACCEPT
	-A PVEFW-FORWARD -p IPv6 -j ACCEPT
	-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
create PVEFW-FWBR-OUT (lAOUWz7/GgaoB9l9aoEGwRdqYU8)
	-A PVEFW-FWBR-OUT -i tap800i0 -j tap800i0-OUT
	-A PVEFW-FWBR-OUT -i tap801i0 -j tap801i0-OUT
	-A PVEFW-FWBR-OUT -i tap802i0 -j tap802i0-OUT
	-A PVEFW-FWBR-OUT -i veth100i0 -j veth100i0-OUT
	-A PVEFW-FWBR-OUT -i veth700i0 -j veth700i0-OUT
create tap800i0-OUT (iv41Vxxe0F4IL50yCpCsxp4i6Ow)
	-A tap800i0-OUT -s ! 3e:66:ee:4e:ba:76 -j DROP
	-A tap800i0-OUT -j ACCEPT
create tap801i0-OUT (GZ+46bW9sPo9jNjvmBFMKIUh7wI)
	-A tap801i0-OUT -s ! be:36:87:2b:1:3a -j DROP
	-A tap801i0-OUT -j ACCEPT
create tap802i0-OUT (bclPyCr1jxyZ4tDlS274bstB2nQ)
	-A tap802i0-OUT -s ! fe:d9:25:40:3e:2e -j DROP
	-A tap802i0-OUT -j ACCEPT
create veth100i0-OUT (T3ApX1bXeEz2QkXj5fbPJ4jFAKw)
	-A veth100i0-OUT -s ! 66:4e:88:f9:f0:a1 -j DROP
	-A veth100i0-OUT -j ACCEPT
create veth700i0-OUT (ex7PX8TAPDSz8RHp+7fkz/y+puE)
	-A veth700i0-OUT -s ! 1a:e7:60:55:5e:1b -j DROP
	-A veth700i0-OUT -j ACCEPT

iptables table raw cmdlist:

ip6tables table raw cmdlist:
detected changes

/var/log/daemon.log

---snip---
Mar 27 19:00:06 pve-pi pvestatd[1284]: QEMU/KVM cannot detect CPU flags on ARM (aarch64)
Mar 27 19:00:06 pve-pi pvestatd[1284]: CPU flag detection failed, will try again after delay
Mar 27 19:00:16 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:00:26 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:00:36 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:00:46 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:00:56 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:01:06 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:01:16 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:01:26 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:01:36 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory
Mar 27 19:01:46 pve-pi pve-firewall[1268]: status update error: unable to open file '/proc/sys/net/bridge/bridge-nf-call-iptables' - No such file or directory

Using PiMox on other SBCs/Server?

I was wondering if I can use this on a Rock64 I own, or be deployed on a ThunderX2 so this could be a very good alternative for ESXi

QEMU / KVM Processors

I've got my homelab set up and working beautifully as a 3-node Pi4/pimox7 cluster (can't thank you enough for putting the project together).

I'd previously had an single-node Proxmox instance running on an ancient HP ProLiant (before catastrophic failure) which made cross-os software development work a breeze. I have been trying to get back to some semblance of the functionality I previously had.

It seems like what's preventing me from spinning up a Win10 VM using kvm64 as the CPU is QEMU missing several of ProMox's usual CPU types. It also looks like the version of QEMU pimox uses isn't the same as what you'd get if you ran apt install -y qemu-system-arm64.

tl;dr I'd really like to spin up an x86 or x86_64 VM, but pimox doesn't seem able to at the moment. What can I do to help resolve that? (For clarity, software development is my day job. When I say help, I mean anything up to and including source-level patches.)

LXC USB Passthrough (CC2531 Zigbee Stick)

I followed this guide: https://forum.proxmox.com/threads/lxc-usb-passthrough-zwave-stick.30058/

On the Host the device shows up:

# lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0451:16a8 Texas Instruments, Inc. CC2531 ZigBee
Bus 001 Device 002: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

# ls -al /dev/bus/usb/001/004
crw-rw-r-- 1 root root 189, 3 Jan  2 16:21 /dev/bus/usb/001/004

ls -al /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 Jan  2 16:21 /dev/ttyACM0

so I added

lxc.cgroup.devices.allow: c 189:* rwm
lxc.mount.entry: /dev/bus/usb/001/004 dev/bus/usb/001/004 none bind,optional,create=dir
lxc.cgroup.devices.allow: c 166:* rwm
lxc.mount.entry: /dev/ttyACM0 dev/ttyACM0 none bind,optional,create=file

to the config.

But in the container i get Error: Operation not permitted, cannot open /dev/ttyACM0, I've created dailout, used root, chmod 777 on /dev/ttyACM0 and privileged container.
But the USB Device is shown by lsusb inside the container.

$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0451:16a8 Texas Instruments, Inc. CC2531 ZigBee
Bus 001 Device 002: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

When I try to add the device with lxc-device I get:

lxc-device add -n 100 /dev/ttyACM0 /dev/ttyACM0
lxc-device: 100: commands.c: lxc_cmd_add_bpf_device_cgroup: 1399 Operation not permitted - Failed to add new bpf device cgroup rule
lxc-device: 100: lxccontainer.c: add_remove_device_node: 4659 set_cgroup_item failed while adding the device node
lxc-device: 100: tools/lxc_device.c: main: 153 Failed to add /dev/ttyACM0 to 100

Could it be that the Kernel is missing a feature or something?

Edit: It should theoretically be possible to passthrough a tty with lxc on a raspberry pi: https://doc.turris.cz/doc/en/public/deconz_lxc_howto (Other os, other stick)

I also tried a USB to UART Stick, with the same result: /dev/ttyUSB0: Operation not permitted

PiMox 7.1 with Kernel Update?

Hey there, first: thank you for your amazing work!

I have serval questions and maybe some ideas to make the PiMox more useful / amazing.

  1. Kernel Update / Modification needed for PiMox 7
    => Proxmox 7 uses KSMTuned, which isnt supported on the default Raspbian OS (Debian 11), you have to modify the Kernel to Support KSM Features, is it possible that you can "ship" the PiMox 7 with the actual kernel latest supported? For Proxmox 7 it would be 5.11 and Proxmox 7.1 it would be 5.13 (afaik)

  2. Update PiMox from 7 to 7.1
    => When we can count on the Update to happen? The new KSM Features are good for CPU/RAM Management specially for the Pi and ARM64 devices (emulated)

  3. Installation of ZRAM instead of SWAP/ZSWAP
    => I Switched from SWAP to ZRAM and with the KSMTuned its an amazing view to see how beautiful the Pi is running right now.
    I have 2 VMs (only) and an M2. SATA SSD Storage for the VMs to run which really speedup things here.

image

I Compiled my own Kernel (5.11.22) with KSM Features activated.

Kind regards,

Use of uninitialized value $path in -c at /usr/share/perl5/PVE/QemuServer.pm line 3691.

Fresh install RpiOS 64BIT LITE version.

Im getting this error when i create and start

Use of uninitialized value $path in -c at /usr/share/perl5/PVE/QemuServer.pm line 3691.
TASK ERROR: no such serial device

Config of vm

balloon: 0
boot: order=virtio0;ide2;net0
cores: 1
ide2: local:iso/debian-11.1.0-amd64-netinst.iso,media=cdrom
memory: 2048
meta: creation-qemu=6.0.0,ctime=1638546622
name: librenms
net0: virtio=9E:E9:A6:AC:20:77,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=8b40d464-a7b1-45f5-bfb0-8b02fb5e1026
sockets: 1
virtio0: local:100/vm-100-disk-0.qcow2,backup=0,cache=writeback,size=10G

Create CT: Internal Server Error: SSH public key validation error

When I try to create an LXC via Ansible I get the response "Error: 500 Internal Server Error: SSH public key validation error".
I also send this request via curl and get exact the same result. If I remove the public key and just enter a password, everything works fine.
I send the same request on an "normal" amd64 installation, and everything works fine.
On both systems The storage is a directory.

The request via curl (Error):
curl -k --request POST --url https://192.168.1.130:8006/api2/extjs/nodes/RPi4-PVE-1/lxc --header 'CSRFPreventionToken: 61CACEDF:PrFwO+0a05XfXy19Ci9SYOVxDIplQQ9kvtOBwlPW/Mg' --header 'Content-Type: application/x-www-form-urlencoded' --header 'Cookie: PVEAuthCookie=PVE%3Aroot@pam%3A61CACEDF%3A%3AJIDTljH2ReIB3IcvLmK+nLF69FRcjFvn7GUKBstJJzNln3Q8ZO/zNUY/0mnimu287k7ICLhYFibuivHW5MUUTJ1bIYkHO9od6qB+SrsKym42dP84WV/EKiwu0+rWANb8ycLWjjz2mySDiesJsFao3JZl8Y2+Zju+qlVe/XWV6dcz+I79rujee0ikKa59MtMFQ1NJVqQX9Km8KoiED50fUVQ2Kmp01mCcllcMarzohu4koBB/88ESphjA4dIdNFfs0NMoS31vI3T8mLCyfhH2Oy7Ce2gtNiZyes3fRX/njuxaIECObVBilzg3Owg3zxabhaMAQVehNzbuWvY+Bjfaxw%3D%3D' --data hostname=test --data ostemplate=local-data:vztmpl/debian-11-standard_11.0-1_arm64.tar.xz --data rootfs=local:8 --data cores=1 --data memory=512 --data swap=512 --data vmid=200 --data ssh-public-keys=ssh-ed25519%20AAAAC3NzaC1lZDI1NTE5AAAAINTE1E65zBpJWr%2BKhOwwOPRXkjsSe49TXT1EcssXcqOu%20andy%40lappi

Request with password (Working):
curl -k --request POST --url https://192.168.1.130:8006/api2/extjs/nodes/RPi4-PVE-1/lxc --header 'CSRFPreventionToken: 61CACEDF:PrFwO+0a05XfXy19Ci9SYOVxDIplQQ9kvtOBwlPW/Mg' --header 'Content-Type: application/x-www-form-urlencoded' --header 'Cookie: PVEAuthCookie=PVE%3Aroot@pam%3A61CACEDF%3A%3AJIDTljH2ReIB3IcvLmK+nLF69FRcjFvn7GUKBstJJzNln3Q8ZO/zNUY/0mnimu287k7ICLhYFibuivHW5MUUTJ1bIYkHO9od6qB+SrsKym42dP84WV/EKiwu0+rWANb8ycLWjjz2mySDiesJsFao3JZl8Y2+Zju+qlVe/XWV6dcz+I79rujee0ikKa59MtMFQ1NJVqQX9Km8KoiED50fUVQ2Kmp01mCcllcMarzohu4koBB/88ESphjA4dIdNFfs0NMoS31vI3T8mLCyfhH2Oy7Ce2gtNiZyes3fRX/njuxaIECObVBilzg3Owg3zxabhaMAQVehNzbuWvY+Bjfaxw%3D%3D' --data hostname=test --data ostemplate=local-data:vztmpl/debian-11-standard_11.0-1_arm64.tar.xz --data rootfs=local:8 --data cores=1 --data memory=512 --data swap=512 --data vmid=200 --data password=asdfg

Any ideas how to debug this issue?
I would prefer public/private key over a password.

can't start a vm (again)

hey, so i tried it again today, but it seems like even tho i choose the desktop iso (64 bit, gui etc) the vnc "plugin" of pimox does not detect any display activity, it just exits in "Guest has not initialized the display (yet)", no matter if i wait a bit, i get a little bit of ram and cpu usage, so it should be doing something, but i don't know what...
ps. i know this is a duplicate of #7 but i can't re-open it due to a collaborator or rather the owner closed it
2021-10-22_11-15 cleaned
there is activity... but no display output

Network connectivity lost after bridge setup

I am having a strange issue after setting this up. I followed the readme and used @TuxfeatMac 's script to update to bullseye and install pimox7. That all worked initially, but I wanted to change my networking to use a bridge interface so I could hook my testing containers/VMs into the bridge. However, after configuring this and rebooting I lost network connectivity to the pimox server (UI and via SSH, ping, etc). I have seen #13 which seems related.

After doing some testing, it seems that for some reason, the real physical network interface gains a local IP (169.254.36.32) and a second default route is created using that within a minute of networking starting (using ifdown and ifup). After resetting the network, I can use it until this second route is made. See the following debug log for further explanation.

Any ideas on what is causing this? I assume it's probably some rogue debian script but I'm not sure how to proceed. Can I intercept calls to route?

$ cat /etc/network/interfaces

auto lo
iface lo inet loopback

iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.55.20.19/24
gateway 10.55.20.1
bridge-ports eth0
bridge-stp off
bridge-fd 0

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether dc:a6:32:8b:4c:1a brd ff:ff:ff:ff:ff:ff
inet 169.254.36.32/16 brd 169.254.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::dea6:168b:ef2a:9b4c/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether dc:a6:32:8b:3a:1d brd ff:ff:ff:ff:ff:ff
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether dc:a6:32:8b:4c:1a brd ff:ff:ff:ff:ff:ff
inet 10.55.20.19/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::dea6:168b:ef2a:9b4c/64 scope link
valid_lft forever preferred_lft forever

$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 0.0.0.0 0.0.0.0 U 0 0 0 eth0
default 10.55.20.1 0.0.0.0 UG 0 0 0 vmbr0
10.55.20.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
link-local 0.0.0.0 255.255.0.0 U 0 0 0 eth0

$ ping 1.1.1.1
doesnt work

$ ping -I vmbr0 1.1.1.1
works

  • cannot access proxmox UI on other PC

manual intervention

$ ifdown vmbr0 eth0

$ ifup vmbr0 eth0

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,DYNAMIC,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether dc:a6:32:8b:4c:1a brd ff:ff:ff:ff:ff:ff
inet6 fe80::dea6:168b:ef2a:9b4c/64 scope link
valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether dc:a6:32:8b:3a:1d brd ff:ff:ff:ff:ff:ff
9: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether dc:a6:32:8b:4c:1a brd ff:ff:ff:ff:ff:ff
inet 10.55.20.19/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::dea6:168b:ef2a:9b4c/64 scope link
valid_lft forever preferred_lft forever

$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.55.20.1 0.0.0.0 UG 0 0 0 vmbr0
10.55.20.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0

$ ping 1.1.1.1
works

  • can access proxmox UI on other PC

after a minute or so, the networking configuration reverts to the situation above the manual intervention line
then network connectivity is lost
for testing purposes, i tried to delete the second default route (eth0) but the route would not delete despite multiple attempts

I know I could remove the bridge and just use masquerading but I want to have full networking

Qemu

I have qemu error if I run a windows vm
Tried a Linux one and I got “screen needs to be configured” I tried q35 machine but I get an ehci error can someone please help

can't start a vm

i'm trying to set up a arch linux arm vm machine, but every time i try starting it i get this error:

kvm: -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101: Bus 'ide.1' not found
TASK ERROR: start failed: QEMU exited with code 1

i tried creating a disk and changed it's "port" to ide 1 but same error message, any ideas on how to start it?

Script Failure 'Syntax error: "(" unexpected (expecting "do")'

I'm getting a syntax error on my raspberry pi cm4 with 4gb of ram. I followed the instructions from the readme and I don't immediately see an additional ( on the line it's reporting. Full error below:

./RPiOS64-IA-Install.sh: 57: Syntax error: "(" unexpected (expecting "do")

This happens as soon as I attempt to enter a hostname. Is there something I may have done with setting up my pi that might cause some issues for this?

RPiOS64bit Interactive Automatic Installer -> Connection failure. Network error or Proxmox VE services not running?

Hi there.
Thanks for the port :-D

i just used the automatic installer on the "2021-10-30-raspios-bullseye-arm64-lite.zip" image on a RPI4.
The Installation runs fine, the WebFrontend is accessible through the network but i can't log in.

With LinuxPAM authentication module and Proxmox VE Authentication
Connection failure. Network error or Proxmox VE services not running?

Any ideas on that? I already tried with the "curl https://raw.githubusercontent.com/pimox/pimox7/master/pimox.sh | sh" in an earlier try on a clean system but i had other issues like VMBR0 not configured or empty dashboard after login in through web.

Installation fails on raspios 64bit

Hello,
i have tried like 20 times to install pimox but the error is always the same:

Done.
Loading new ceph-dkms-0.0.2 DKMS files...
It is likely that 5.15.30-v8+ belongs to a chroot's host
Building for 5.15.30-v8+
Building initial module for 5.15.30-v8+
Error! Bad return status for module build on kernel: 5.15.30-v8+ (aarch64)
Consult /var/lib/dkms/ceph-dkms/0.0.2/build/make.log for more information.
dpkg: error processing package ceph-dkms (--configure):
installed ceph-dkms package post-installation script subprocess returned error exit status 10
Setting up pve-manager (7.1-10) ...
Job for pvestatd.service failed because the control process exited with error code.
See "systemctl status pvestatd.service" and "journalctl -xe" for details.
dpkg: error processing package pve-manager (--configure):
installed pve-manager package post-installation script subprocess returned error exit status 1
Setting up zfs-dkms (2.0.3-9) ...
Removing old zfs-2.0.3 DKMS files...


Deleting module version: 2.0.3
completely from the DKMS tree.

Done.
Loading new zfs-2.0.3 DKMS files...
It is likely that 5.15.30-v8+ belongs to a chroot's host
Building for 5.15.30-v8+
Building initial module for 5.15.30-v8+

configure: error:
*** None of the expected "capability" interfaces were detected.
*** This may be because your kernel version is newer than what is
*** supported, or you are using a patched custom kernel with
*** incompatible modifications.
***
*** ZFS Version: zfs-2.0.3-9
*** Compatible Kernels: 3.10 - 5.10

Error! Bad return status for module build on kernel: 5.15.30-v8+ (aarch64)
Consult /var/lib/dkms/zfs/2.0.3/build/make.log for more information.
dpkg: error processing package zfs-dkms (--configure):
installed zfs-dkms package post-installation script subprocess returned error exit status 10
dpkg: dependency problems prevent configuration of proxmox-ve:
proxmox-ve depends on pve-manager; however:
Package pve-manager is not configured yet.
proxmox-ve depends on ceph-dkms; however:
Package ceph-dkms is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of zfs-zed:
zfs-zed depends on zfs-modules | zfs-dkms; however:
Package zfs-modules is not installed.
Package zfs-dkms which provides zfs-modules is not configured yet.
Package zfs-dkms is not configured yet.

dpkg: error processing package zfs-zed (--configure):
dependency problems - leaving unconfigured
Processing triggers for initramfs-tools (0.140) ...
Errors were encountered while processing:
ceph-dkms
pve-manager
zfs-dkms
proxmox-ve
zfs-zed
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@rasp1-prox:~#

I followed every single step in this guide (and i tried even other guides) but the installation in never "successful".

My hardware:

Raspberri Pi 4 model B 8GB RAM
32GB Samsung SD Card

Tried even the os image described in the installation script but nothing changed, the error is still the same.
I really don't know what to do next or what to try :/

unmet dependencies

root@raspberrypi:/home/pi# curl https://raw.githubusercontent.com/pimox/pimox7/master/pimox.sh | sh
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100   364  100   364    0     0   4333      0 --:--:-- --:--:-- --:--:--  4385
Reading package lists... Done
Building dependency tree
Reading state information... Done
gnupg is already the newest version (2.2.12-1+deb10u1).
The following package was automatically installed and is no longer required:
python-colorzero
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  3143  100  3143    0     0  34163      0 --:--:-- --:--:-- --:--:-- 34163
OK
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian-security buster/updates InRelease
Hit:3 http://archive.raspberrypi.org/debian buster InRelease
Hit:4 http://deb.debian.org/debian buster-updates InRelease
Hit:5 http://deb.debian.org/debian bullseye InRelease
Get:6 https://raw.githubusercontent.com/pimox/pimox7/master dev/ InRelease [2,186 B]
Fetched 2,186 B in 2s (1,314 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
proxmox-ve : Depends: pve-manager but it is not going to be installed
Depends: pve-qemu-kvm but it is not going to be installed
Depends: qemu-server but it is not going to be installed
Depends: spiceterm but it is not going to be installed
Depends: vncterm but it is not going to be installed
Depends: zfsutils-linux but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

updated the os first, did the other steps too but it seems like there are unresolved dependencies, does anyone know what i could do so it installs?

The docker on Host OS is conflict to guest os network

After installing a docker on host OS(Debian11), my guest OS(Debian10) network was stop.
Oh the oher hand, docker install on guest OS(Debina10) didn't affect to host OS.
After remove docker on host OS, guest os is back to normal.
Is there any special method to use docker on Host OS?

Guest has not initialized display yet

After update with #31 i started VM but getting info - guest has not initialized display yet. VM load is still @100% and nothing happens

Config is without disk, even bios not loading :(

agent: 0
balloon: 0
boot: order=net0
cores: 1
memory: 2048
meta: creation-qemu=6.0.0,ctime=1638565898
name: librenms
net0: virtio=8E:B8:7E:E9:00:1C,bridge=vmbr0
numa: 0
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=e504fca4-15ae-49b6-9253-645c1c4e6fc5
sockets: 1

When changed config to. VM start loading and seems netinst iso is loading. But i dont want EFI boot :/

agent: 0
balloon: 0
bios: ovmf
boot: order=net0;scsi2;virtio0
cores: 1
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=64M
memory: 2048
meta: creation-qemu=6.0.0,ctime=1638565898
name: librenms
net0: virtio=8E:B8:7E:E9:00:1C,bridge=vmbr0
numa: 0
ostype: l26
scsi2: local:iso/debian-11.1.0-arm64-netinst.iso,media=cdrom,size=327980K
scsihw: virtio-scsi-pci
smbios1: uuid=e504fca4-15ae-49b6-9253-645c1c4e6fc5
sockets: 1
virtio0: local:100/vm-100-disk-1.qcow2,size=10G

QEMU/KVM cannot detect CPU flags on ARM (aarch64) ?

I see these over and over. Are these ok considering the "hacky" nature of pimox?

Jan 13 22:42:45 RPi4-PVE pvestatd[1081]: CPU flag detection failed, will try again after delay
Jan 13 22:42:46 RPi4-PVE ksmtuned[503]: /usr/sbin/ksmtuned: line 61: /sys/kernel/mm/ksm/run: No such file or directory
Jan 13 22:43:47 RPi4-PVE ksmtuned[503]: /usr/sbin/ksmtuned: line 61: /sys/kernel/mm/ksm/run: No such file or directory
Jan 13 22:44:45 RPi4-PVE pvestatd[1081]: QEMU/KVM cannot detect CPU flags on ARM (aarch64)
Jan 13 22:44:45 RPi4-PVE pvestatd[1081]: CPU flag detection failed, will try again after delay
Jan 13 22:44:47 RPi4-PVE ksmtuned[503]: /usr/sbin/ksmtuned: line 61: /sys/kernel/mm/ksm/run: No such file or directory
Jan 13 22:45:47 RPi4-PVE ksmtuned[503]: /usr/sbin/ksmtuned: line 61: /sys/kernel/mm/ksm/run: No such file or directory
Jan 13 22:46:45 RPi4-PVE pvestatd[1081]: QEMU/KVM cannot detect CPU flags on ARM (aarch64)

At the time NO vm or ct was running for quite a while.

Thanks for the hard work! Its amazing.

Running on RPi 3B+ !!!

First of all thanks for your work! This is absolutely amazing!
So far it is perfectly running fine on my RPi4.

I have some "old" modded RPi3B+ witch I think I can't sell (removed display/camera/audio connectors for a heat spreader & mount plate).... I don't have a use case for them anymore, so I gave it a try... =)

I saw on my RPi4 installation that the idle memory consumption is around 1GB. witch could be a problem, since the RPi3 only has 1GB RAM available...
I used zram and dphys-swapfile to add another extra 1,25GB "mem", a overclock to 1,4GHz and it worked!

xterm jsFailRPi3

But until now I'm unable to get a VM or CT running, also I cannot connect to the shell of the RPi via the WebGUI, but NoVNC works fine as you can see from the screenshot.

I know the requirement is a RPi4.
So my questions are:
Is there any way to get involved in this what you have done, so that I can maybee adopt it for the RPi3?
Is there any way how to support you?

error with pimox.sh

error like this :

The following information may help to resolve the situation:

The following packages have unmet dependencies:
 proxmox-ve : Depends: pve-manager but it is not installable
              Depends: pve-qemu-kvm but it is not installable
              Depends: qemu-server but it is not installable
              Depends: spiceterm but it is not installable
              Depends: vncterm but it is not installable
E: Unable to correct problems, you have held broken packages.

Question about port

Hello,

First thank you for your great and amazing work !
But I'm really curious, how did you do to do the port of Proxmox to RPI ? I don't understand how did you proceed ?
Thank you very much for your explanation.

Best regards,

High CPU Load after booting an LXC container

Hi,

I'm often getting about 33% CPU load after booting an LXC container. Booting more LXC containers will lead to a compounding increase in CPU load, I've seen up to about 66% total load even though the heaviest workload was a mere pihole.

Processes with high CPU utilization are udisksd and systemd-udevd.

Setting a kill command on the udisksd process will instantly get rid of the systemd-udevd processes as well and leave CPU utilization at a far more sane 1-3% while idling.

The LXC containers themselves do not report any significant CPU utilization.
Using the kill command does not seem to negatively affect any processes. Certainly Proxmox and the containers keep right on ticking.

Background info:
Raspberry Pi 4 8GB
Linux 5.10.103-v8+ #1530
pve-manager/7.1-10/821f136a
Jan28 release of Raspberry Pi OS Lite 64-bit and most recent pimox7 version as of ~2 weeks ago
CT template source: file: https://uk.lxd.images.canonical.com/images/ubuntu/focal/arm64/default/20220318_07:43/rootfs.tar.xz (not available anymore)

Not exactly a big issue since there's an easy workaround, but I'm pretty sure that something isn't working as intended.

Best regards,
steamrick

Operation not permitted while setup, cannot run lxc/vm

I download latest bullseye 64bit and follow the instructions. After restart, th

pi@pimox:~ $ sudo -s

root@pimox:/home/pi# apt upgrade -y

[...]
ln: failed to create symbolic link '/boot/pve/vmlinuz': Operation not permitted
ln: failed to create symbolic link '/boot/pve/initrd.img': Operation not permitted
[...]

I can login into gui, the summary looks like that:
Screenshot 2022-03-10 at 20 26 20

running containers fails with:

sync_wait: 36 An error occurred in another process (expected sequence number 7)
__lxc_start: 2073 Failed to spawn container "101"
TASK ERROR: startup for container '101' failed

These are basic debian11 and alpine templates downloaded from CT Templates
Screenshot 2022-03-10 at 20 31 47

I cannot make it to work, tried googling but nothing relevant pops out.

Can't create an EFI disk anymore

Hi there,
I use proxmox on raspberry pi 4 (thus on arm64)

So far everything has been going well. A few days ago I performed apt update and apt upgrade on the hosts.

Now I wanted to create a VM as usual and found that the following error occurs.

"Can't use an undefined value as an ARRAY reference at /usr/share/perl5/PVE/QemuServer.pm line 3173."

I have found out so far that it appears to be a UEFI BIOS related problem.
Specifically, the error occurs when creating the EFI disk. (Without it works)

I hope you have an idea or let me know which logs / information I should leave out.

Many greetings

Take snapshot vm failed

Hi all,

Anyone can help why I can not take a snapshot vm. The error messages below.

TASK ERROR: VM 100 qmp command 'query-savevm' failed - got timeout

image

USB Passthrough works (Atheros WIFI) but randomly resets and dies..

Anyone else has random resets on USB devices?

Everything works like a charm (USB 2.0, USB 3 NOT selected) until there is a random reset.

root@RPi4-PVE:~# uname -a
Linux RPi4-PVE 5.10.63-v8+ #1488 SMP PREEMPT Thu Nov 18 16:16:16 GMT 2021 aarch64 GNU/Linux

lsusb

Bus 001 Device 011: ID 0cf3:9271 Qualcomm Atheros Communications AR9271 802.11n

pimox dmesg

		[ 1201.176990] usb 1-1.1: ath9k_htc: USB layer deinitialized
		[ 1211.438991] usb 1-1.1: reset high-speed USB device number 6 using xhci_hcd
		[26465.440460] vmbr0: port 4(tap105i0) entered disabled state
		[26465.575549] usb 1-1.1: reset high-speed USB device number 6 using xhci_hcd
		[26470.891366] usb 1-1.1: device descriptor read/64, error -110
		[26486.507421] usb 1-1.1: device descriptor read/64, error -110
		[26486.695427] usb 1-1.1: reset high-speed USB device number 6 using xhci_hcd
		[26491.883378] usb 1-1.1: device descriptor read/64, error -110
		[26507.499569] usb 1-1.1: device descriptor read/64, error -110
		[26507.687559] usb 1-1.1: reset high-speed USB device number 6 using xhci_hcd
		[26509.311786] usb 1-1.1: Device not responding to setup address.
		[26511.147381] usb 1-1.1: Device not responding to setup address.
		[26511.355328] usb 1-1.1: device not accepting address 6, error -71
		[26511.435341] usb 1-1.1: reset high-speed USB device number 6 using xhci_hcd
		[26513.059355] usb 1-1.1: Device not responding to setup address.
		[26514.891381] usb 1-1.1: Device not responding to setup address.
		[26515.099417] usb 1-1.1: device not accepting address 6, error -71
		[26515.106829] usb 1-1.1: USB disconnect, device number 6
		[26515.187404] usb 1-1.1: new high-speed USB device number 7 using xhci_hcd
		[26520.299629] usb 1-1.1: device descriptor read/64, error -110
		[26535.915757] usb 1-1.1: device descriptor read/64, error -110
		[26536.103541] usb 1-1.1: new high-speed USB device number 8 using xhci_hcd
		[26541.291731] usb 1-1.1: device descriptor read/64, error -110

inside the VM, all working perfect until...

		[  101.433151] wlan0: authenticate with 00:2a:10:cf
		[  103.328896] wlan0: send auth to 00:2a:10:cf (try 1/3)
		[  103.335886] wlan0: authenticated
		[  103.340161] wlan0: associate with 00:2a:10:cf (try 1/3)
		[  103.347913] wlan0: RX AssocResp from 00:2a:10:cf (capab=0x431 status=0 aid=12)
		[  103.421820] wlan0: associated
		[  103.539850] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
		[18087.696891] ath: phy0: Unable to remove station entry for: 00:2a:10:cf
		[22266.666353] ath: phy0: Chip reset failed
		[22266.667579] ath: phy0: Unable to reset channel (2447 Mhz) reset status -22
		[22266.669590] ath: phy0: Unable to set channel

Can't connect to network at sub OS(debian 10)

I have installed debain 10 at pimox7.
But debian 10 can't connect to network.
I tried several normal solution for network problem but failed.

my-debian10_after_install_network

this is '/etc/network/interface' at debian11

auto lo
iface lo inet loopback

auto enabcm6e4ei0
iface enabcm6e4ei0 inet static
	address 192.168.35.163/24
	gateway 192.168.35.1

auto vmbr0
iface vmbr0 inet static
        address 192.168.35.8/24
        gateway 192.168.35.1
	bridge-ports none      # If change it to enabcm6e4ei0, debian11 remote connection is disconnected
	bridge-stp off
	bridge-fd 0

And this is '/etc/network/interface' at debian10

auto enp0s18
iface enp0s18 inet static
        address  192.168.35.8/24
        gateway 192.168.35.1

How can I connect to network? Any Idea?

VLAN support on pimox7

Hi,

given by the fact 'vlan' package isn't installed by 'RPiOS64-IA-Install.sh' script I tried to do it, but this will remove 'proxmox-ve' and 'pve-manager' packages...
I need to add VLAN's on my eth0 for untagged interfaces on CT's, as normal I think...

Best.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.