amboar / culvert Goto Github PK

Hi,

I've encountered failure in probing the interfaces via in-band BMC from the host. Are there any pre-requisites that I may be missing before I start the probing?

[:~/culvert]# ./build/src/culvert probe --list-interfaces
[] failed to initialise devmem bridge: -1
[] Failed to probe SoC revision: -19
[*] Failed to probe SoC, exiting: -19

Is there a support list I can refer to check which SOCs are supported by this utility?

After a very silly mixup of which host I was on I accidentally ran two instances of culvert in parallel. One was flashing the firmware and the other triggered a WDT reset. This left my system in a pretty broken state! Fortunately this was in my lab so it was easily rectified, but it made me wonder if culvert could do something to prevent multiple instances from running in parallel?

I assume the right way to do this is with flock? I'd probably just wrap the entire program (somewhere in https://github.com/amboar/culvert/blob/main/src/culvert.c?) instead of specific subcommands, that would prevent my scenario from occurring again.

I'm open to attempting this myself, but wanted to see if anyone else had opinions or thinks this is a terrible idea before I look at writing any code!

I have motherboard Supermicro X11SSL-F.
I have compiled an openbmc firmware and am trying to flash this motherboard with it. The discord channel Openbmc advised me to use your tool, but I'm definitely not qualified to figure out what I'm doing wrong. Tell me what can be done about it?

root@hostname ~ # uname -r
6.1.8-arch1-1

Iculvert: v0.4.0-119-g5f4db2b

root@hostname~ # ./culvert --verbose probe
[] Found 5 registered bridge drivers
[] Trying bridge driver Debug UART
[] Unrecognised argument list for debug interface (0)
[] Trying bridge driver devmem
[] failed to initialise devmem bridge: -1
[] Trying bridge driver iLPC2AHB
[] Probing iLPC2AHB
[] Probing 0x2e for SuperIO
[] Found SuperIO device at 0x2e
[] Probing for SoC revision registers
[] Found revision 0x2010303
[] Trying bridge driver LPC2AHB
[] Failed to initialise L2A bridge: -95
[] Trying bridge driver P2A
[] Probing P2A
[] Probing for SoC revision registers
[] Found revision 0x2010303
[] Probing for SoC revision registers
[] Found revision 0x2010303
[] Selected devicetree for SoC 'aspeed,ast2400'
[] Found 14 registered drivers
[] Bound sdmc driver to /ahb/apb/memory-controller@1e6e0000
[] Bound strap driver to /ahb/apb/syscon@1e6e2000/strapping
[] Bound sioctl driver to /ahb/apb/syscon@1e6e2000/superio
[] Bound pciectl driver to /ahb/apb/syscon@1e6e2000/bridge-controller
[] Bound vuart driver to /ahb/apb/serial@1e787000
[] Bound ilpcctl driver to /ahb/apb/lpc@1e789000/bridge-controller
[] Initialised strap driver
[] Initialised sioctl driver
[] Initialised ilpcctl driver
[] Initialised ilpcctl AHB bridge controller
[] fdt: Searching devicetree for type 'memory'
[] Initialised sdmc driver
[] Initialised pciectl driver
[] Initialised pciectl AHB bridge controller
xdma: Permissive
BMC: Disabled
VGA: Enabled
XDMA on VGA: Enabled
XDMA is constrained: No
p2a: Permissive
BMC: Disabled
VGA: Enabled
MMIO on VGA: Enabled
[0x00000000 - 0x17ffffff] Firmware: Writable
[0x18000000 - 0x1fffffff] SoC IO: Writable
[0x20000000 - 0x2fffffff] BMC Flash: Writable
[0x30000000 - 0x3fffffff] Host Flash: Writable
[0x40000000 - 0x5fffffff] DRAM: Writable
[0x60000000 - 0x7fffffff] LPC Host: Writable
[0x80000000 - 0xffffffff] Reserved: Writable
ilpc: Permissive
SuperIO address: 0x2e
[] Unbound instance of driver ilpcctl
[] Unbound instance of driver vuart
[] Unbound instance of driver pciectl
[] Unbound instance of driver sioctl
[] Unbound instance of driver strap
[*] Unbound instance of driver sdmc

when i try flashing bmc:
./culvert write firmware < ./path/to/openbmc.mtd
I get an error:
[*] failed to initialise devmem bridge: -1

When doing the same steps for x11spi, the platform crashes.

Hi, this may be a long shot, but I've got a Gigabyte MZ32-AR0 rev3 with a BMC that's not starting up (steady BMC_LED light, no firmware version displayed during POST, no devices detected by ipmi_si). I came across this utility and I was wondering if I could leverage this to possibly factory reset it and get it back to a working state. It looks like the BMC stopped working after I put the board in BIOS recovery mode and I've spent countless hours trying to get it back up. Connecting to the server's serial port gave me no more useful info and connecting to BMC_UART only got me this:

DRAM Init-V12-DDR4
0abc1-4Gb-Done
Read margin-DL:0.3862/DH:0.3941 CK (min:0.30)


U-Boot 2013.07 (Nov 01 2023 - 17:52:22)

I2C:   ready
DRAM:  424 MiB
eSPI Handshake complete
OEM_BOARD_INIT - Start (BMC)
LPC mode
OEM_BOARD_INIT - End
Flash: ERROR: Unable to Detect SPI Flash
*** failed ***
### ERROR ### Please RESET the board ###

I've tried flashing a new FW (a supported one of course, from the mobo's support page) to no avail (it flashes "correctly" but I still can't detect the BMC, and the mgmt port doesn't appear to be in a proper state based on the flashing). I've tried connecting to the console using this tool but that didn't get me anywhere either.

(meson) root@pve:~/culvert/build/src# ./culvert -v console uart3 uart2 115200 admin *****************
[*] Found 5 registered bridge drivers
[*] Trying bridge driver l2a
[*] Failed to initialise L2A bridge: -95
[*] Trying bridge driver ilpc
[*] Probing ilpc
[*] Probing 0x2e for SuperIO
[*] Found SuperIO device at 0x2e
[*] Probing for SoC revision registers
[*] Found revision 0x4030303
[*] Trying bridge driver devmem
[*] failed to initialise devmem bridge: -1
[*] Trying bridge driver debug-uart
[*] Unrecognised argument list for debug interface (0)
[*] Trying bridge driver p2a
[*] Probing p2a
[*] Probing for SoC revision registers
[*] Found revision 0x4030303
[*] Probing for SoC revision registers
[*] Found revision 0x4030303
[*] Selected devicetree for SoC 'aspeed,ast2500'
[*] Found 15 registered drivers
[*] Bound trace driver to /ahb/bus-controller@1e600000
[*] Bound sfc driver to /ahb/apb/spi@1e620000
[*] Bound sfc driver to /ahb/apb/spi@1e630000
[*] Bound sfc driver to /ahb/apb/spi@1e631000
[*] Bound sdmc driver to /ahb/apb/memory-controller@1e6e0000
[*] Bound clk driver to /ahb/apb/syscon@1e6e2000/clock
[*] Bound strap driver to /ahb/apb/syscon@1e6e2000/strapping
[*] Bound sioctl driver to /ahb/apb/syscon@1e6e2000/superio
[*] Bound bridge-controller driver to /ahb/apb/syscon@1e6e2000/bridge-controller
[*] Bound debugctl driver to /ahb/apb/syscon@1e6e2000/debug-bridge-controller
[*] Bound pciectl driver to /ahb/apb/syscon@1e6e2000/pcie-bridge-controller
[*] Bound scu driver to /ahb/apb/syscon@1e6e2000
[*] Bound wdt driver to /ahb/apb/watchdog@1e785000
[*] Bound wdt driver to /ahb/apb/watchdog@1e785020
[*] Bound wdt driver to /ahb/apb/watchdog@1e785040
[*] Bound vuart driver to /ahb/apb/serial@1e787000
[*] Bound ilpcctl driver to /ahb/apb/lpc@1e789000/bridge-controller
[*] Bound uart-mux driver to /ahb/apb/lpc@1e789000
[*] Initialised scu driver
[*] Initialised clk driver
[*] Initialised uart-mux driver
[*] Enabling UART clocks
[*] Routing UART3 to UART5
[*] Initialising SUART3
[*] SUART base address: 0x3e8
[*] SUART SIRQ: 6
[*] Configuring baud rate of 115200 for BMC console
[*] Starting getty from BMC console
[*] Launched getty with: /sbin/agetty -8 -L ttyS1 1200 xterm &
[*] Routing UARTs to connect UART3 with UART2
[*] Setting target baud rate of 115200

At this point I'm running out of options and I've only got until Friday to return the board (which will take a few weeks) so I'm hoping for a hail mary at this point. I went through the issues and the little documentation I could find about this tool and didn't get anywhere. Do you have anything to suggest?

This is what I see when probing:

debug:	Permissive
	Debug UART port: UART5
xdma:	Restricted
	BMC: Disabled
	VGA: Enabled
	XDMA on VGA: Enabled
	XDMA is constrained: Yes
p2a:	Permissive
	BMC: Disabled
	VGA: Enabled
	MMIO on VGA: Enabled
	[0x00000000 - 0x0fffffff]   Firmware: Writable
	[0x10000000 - 0x1fffffff]     SoC IO: Writable
	[0x20000000 - 0x2fffffff]  BMC Flash: Writable
	[0x30000000 - 0x3fffffff] Host Flash: Writable
	[0x40000000 - 0x5fffffff]   Reserved: Writable
	[0x60000000 - 0x7fffffff]   LPC Host: Writable
	[0x80000000 - 0xffffffff]       DRAM: Writable
ilpc:	Permissive
	SuperIO address: 0x2e

Reading the RAM seems to get me info similar to what I'm getting from u-boot

(meson) root@pve:~/culvert/build/src# ./culvert read ram 
[*] failed to initialise devmem bridge: -1
[*] 512MiB DRAM with 32MiB VRAM; dumping 480MiB (0x80000000-0x9dffffff)
t<�T=�t<�������T=�t<�
в424T=�_���������>�й
��������T=��=�l���=�V=�(%P%�|>�424�=�(%P%�u/���>� MiBoot 2013.07 (Nov 01 2023 - 17:52:22)

�, ?���� ?����u/����]���w�M�u/�K�u/��I�,�0�K��k������n���$S�k��~k��n��k�0x1c200�����@�@.........

Thanks!

[zev@hatter: culvert]% git describe
v0.4.0-135-g3b981efbaf80
[zev@hatter: culvert]% meson compile -C build
...
[zev@hatter: culvert]% ./build/src/culvert | head -n1
culvert: v0.4.0-135-g3b981efbaf80
[zev@hatter: culvert]% git commit --amend -CHEAD
...
[zev@hatter: culvert]% git describe
v0.4.0-135-g5be169694ed7
[zev@hatter: culvert]% meson compile -C build
...
[zev@hatter: culvert]% ./build/src/culvert | head -n1
culvert: v0.4.0-135-g3b981efbaf80
[zev@hatter: culvert]% 

Currently it's auto-appended to config.h, but only referenced in src/culvert.c. Maybe to avoid spurious rebuilds it should go into a dedicated version.h header that only gets included where it's needed?

(I can probably hack this together myself, mostly just posting here as a reminder so it doesn't get forgotten, in case anyone else feels like taking care of it before I get around to it, and to potentially get input on implementation details for an appropriate fix.)

Latest version meet ilpcb_init: Operation not supported in 4.14.49-8.ppc64le,

compatiable problem or correct response with OEM patch for cve-2019-6260?

[root@64 ~/cve-2019-6260-master]# ls /sys/kernel/debug/powerpc/lpc/io -lh
-rw------- 1 root root 0 Jul 15 2020 /sys/kernel/debug/powerpc/lpc/io
[root@64 ~/cve-2019-6260-master]# strace ./doit -vv ilpc read 0
execve("./doit", ["./doit", "-vv", "ilpc", "read", "0"], [/* 24 vars */]) = 0
brk(0) = 0x34210000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=38779, ...}) = 0
mmap(NULL, 38779, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fff96b70000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\25\0\1\0\0\0@G\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2146360, ...}) = 0
mmap(NULL, 1922488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fff96990000
mmap(0x7fff96b50000, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b0000) = 0x7fff96b50000
close(3) = 0
mprotect(0x7fff96b50000, 65536, PROT_READ) = 0
mprotect(0x10020000, 65536, PROT_READ) = 0
mprotect(0x7fff96bd0000, 65536, PROT_READ) = 0
munmap(0x7fff96b70000, 38779) = 0
dup(2) = 3
fcntl(3, F_GETFL) = 0x10002 (flags O_RDWR|0x10000)
brk(0) = 0x34210000
brk(0x34240000) = 0x34240000
brk(0) = 0x34240000
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fff96b70000
write(3, "ilpcb_init: Operation not suppor"..., 36ilpcb_init: Operation not supported
) = 36
close(3) = 0
munmap(0x7fff96b70000, 4096) = 0
exit_group(1) = ?
+++ exited with 1 +++
[root@64 ~/cve-2019-6260-master]# uname -r
4.14.49-8.ppc64le
[root@64 ~/cve-2019-6260-master]#

I'm attempting to build on an ARM (Raspberry Pi) system but hitting an error:

$ meson setup build                                                                                                               
The Meson build system
Version: 0.56.1
Source dir: /home/pi/src/doit
Build dir: /home/pi/src/doit/build
Build type: native build
Project name: cve-2019-6260
Project version: v0.3.0
C compiler for the host machine: cc (gcc 8.3.0 "cc (Raspbian 8.3.0-6+rpi1) 8.3.0")                                                                            
C linker for the host machine: cc ld.bfd 2.31.1
Host machine cpu family: arm
Host machine cpu: armv7l
Found pkg-config: /usr/bin/pkg-config (0.29)
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency libfdt found: NO (tried pkgconfig and cmake)
Looking for a fallback subproject for the dependency libfdt

|Executing subproject dtc method meson
|
|Project name: dtc
|Project version: 1.6.0
|C compiler for the host machine: cc (gcc 8.3.0 "cc (Raspbian 8.3.0-6+rpi1) 8.3.0")                                                                           
|C linker for the host machine: cc ld.bfd 2.31.1
|Compiler for C supports arguments -Wpointer-arith: YES
|Compiler for C supports arguments -Wcast-qual: YES
|Compiler for C supports arguments -Wnested-externs: YES
|Compiler for C supports arguments -Wstrict-prototypes: YES
|Compiler for C supports arguments -Wmissing-prototypes: YES
|Compiler for C supports arguments -Wredundant-decls: YES
|Compiler for C supports arguments -Wshadow: YES
|Run-time dependency yaml-0.1 found: YES 0.2.1
|Run-time dependency valgrind found: YES 3.7.0
|Program python3 found: YES (/usr/bin/python3)
|Program swig found: YES (/usr/bin/swig)
|Found git repository at /home/pi/src/doit/subprojects/dtc
|Compiler for C supports link arguments -Wl,--version-script=/home/pi/src/doit/subprojects/dtc/libfdt/version.lds: YES                                        
|Program flex found: YES (/usr/bin/flex)
|Program bison found: YES (/usr/bin/bison)
|Check usable header "fnmatch.h" : YES
|Library dl found: YES
|Program run_tests.sh found: YES (/home/pi/src/doit/subprojects/dtc/tests/run_tests.sh)                                                                       
|Build targets in project: 80
|Subproject dtc finished.

Dependency libfdt from subproject subprojects/dtc found: YES 1.6.0
Dependency dtc from subproject subprojects/dtc found: YES 1.6.0

src/meson.build:57:14: ERROR: Expecting rparen got string.
if fs.is_dir(f'arch/@host@')
              ^

A full log can be found at /home/pi/src/doit/build/meson-logs/meson-log.txt

(The full meson-log.txt in case that's useful, though I'm guessing it's probably irrelevant.)

it looks like not support for apseed 2600?

How to properly verify In-band BMC console from the host?
Runing doit probe is enough?

[~/cve-2019-6260-master]# ./doit probe [*] Probing AHB interfaces [*] Probes failed, cannot access BMC AHB

My server has a ast2500 BMC and AMD ROME SOC

Hi Andrew!

Thanks for reaching out on Twitter, this looks exactly like the thing I'm trying to do. I'm really in awe of what you've done in Culvert so far, and I have a feeling I'll have a lot more questions in the future! Some background:

• fwupd is a system daemon focussed on firmware updates and platform security, and gets metadata and new firmware from the LVFS, another one of the connected projects I maintain. fwupd is installed by default on basically every Linux distro, and also included in ChromeOS -- and the LVFS has supplied over 77 million updates -- so we're doing okay. :)
• HSI is shorthand for the Host Security ID, which is a set of tests we run on the machine. We're checking for UEFI Secure Boot, the IOMMU being set up correctly, things like various BootGuard straps being configured correctly, and dozens of things more. It's essentially what you've done with Culvert but instead focussing on mainly-consumer Intel/AMD UEFI and some aarch64 bits and getting buy-in from vendors about actually fixing the issues and dragging up the level of security for the platform. It's all documented here: https://fwupd.github.io/libfwupdplugin/hsi.html and it's being expanded all the time with new tests.
• The fwupd HSI tests are being run at scale with deep integration with the desktop done, Red Hat insights done, and also openSCAP planned as well. We're working with various large, ahem, companies making HSI compliance a bit part of commercial purchasing agreements.
• A mega vendor want to run fwupd on the BMC itself, initially targetting AST2500 and AST2600 -- and are concerned that OpenBMC/uboot might not doing all it should do. This is why I'm now poking around with the datasheet working out if anything needs to be verified (it does) and if anything exists already (it seems it does!). Several other vendors are using AST2X00 devices and want to verify the BMC security from the host, as the server proprietary BMC stacks are somewhat fragile.

Anyway, this issue is too long already, and is probably actually a discussion -- so if you'd like to move this to email or even a quick zoom call let me know. I'm working for Red Hat in the UK if that helps. Thanks for any ideas or feedback.