Rainbow screen with some SD cards on 3A+, 3B, 3B+ but not Pi 2 (Was: v1.30 doesn't boot past rainbow on 3A+, 3B, 3B+. Works on Pi 2 v1.2) about rpi3 HOT 12 CLOSED

Well, I know it's been a LONG time since I chimed in on this, but I just encountered it again.

I've been using u-boot instead of UEFI for the past year or so, and didn't run into this issue. But I recently dusted some stuff off and tried to get UEFI working again. And I hit this problem again, on two different SDs: a Samsung 64 EVO Plus and a KOOTION 64 GB (bought off Amazon). The Samsung booted once (from cold boot) but would not warm boot afterward. The KOOTION failed each time I tried.

So I copied the u-boot.bin and adjusted the config.txt file to use it instead of RPI_EFI.fd. I did 5 warm reboots in a row without failure.

Then I tweaked the config.txt file back to load RPI_EFI.fd, and it failed again on the first time.

I feel somewhat confident that the problem is in UEFI's handling of the RPI 3B+ SD interface. I've never seen u-boot fail with these cards. And now I seem to be able to reproduce the failure with UEFI regularly, while never reproducing it with u-boot.

from rpi3.

Crazy thing. I went back to 1.27 and it wasn't working and I know for a fact that release worked.

The SD card. No explanation yet, but my "Samsung EVO 32Gib" doesn't work in 3A+/3B/3B+, while SanDisk Ultra Plus does.

from rpi3.

This issue seems to have the same symptoms as #17

from rpi3.

Also affects my Samsung EVO 32GB on RPi UEFI v1.36.

Related serial log:

Raspberry Pi Bootcode
Read File: config.txt, 169
Read File: start.elf, 2959904 (bytes)
Read File: fixup.dat, 7278 (bytes)
MESS:00:00:01.032649:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.036988:0: brfs: File read: 169 bytes
MESS:00:00:01.122120:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.126407:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:01.950083:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:01.957625:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:01.963549:0: *** Restart logging
MESS:00:00:01.967426:0: brfs: File read: 169 bytes
MESS:00:00:02.004874:0: HDMI0: hdmi_pixel_encoding: 162000000
MESS:00:00:02.014248:0: dtb_file 'bcm2710-rpi-3-b.dtb'
MESS:00:00:02.021993:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b.dtb
MESS:00:00:02.026649:0: Loading 'bcm2710-rpi-3-b.dtb' to 0x1f0000 size 0x70fb
MESS:00:00:02.047101:0: brfs: File read: 28923 bytes
MESS:00:00:02.118921:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:02.128989:0: brfs: File read: 169 bytes
MESS:00:00:02.132242:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:02.152852:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:02.838356:0: brfs: File read: /mfs/sd/RPI_EFI.fd
MESS:00:00:02.842213:0: Loading 'RPI_EFI.fd' to 0x0 size 0x1f0000
MESS:00:00:02.848037:0: No compatible kernel found
MESS:00:00:02.852539:0: Device tree loaded to 0x1f0000 (size 0x7588)
MESS:00:00:02.859617:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:02.866306:0: uart: Baud rate change done...
MESS:00:00:02.869720:0: uart: Baud rateNOTICE:  Booting Trusted Firmware
NOTICE:  BL1: v2.5(release):
NOTICE:  BL1: Built : 18:50:24, Jun 14 2021
NOTICE:  rpi3: Detected: Raspberry Pi 3 Model B (1GB, Sony, UK) [0x00a02082]
NOTICE:  BL1: Booting BL2
ERROR:   rpi3_sdhost: unknown err, cmd = 0xd
ERROR:   rpi3_sdhost status: 0x10
ERROR:   rpi3_sdhost: unknown err, cmd = 0xd
ERROR:   rpi3_sdhost status: 0x10
ERROR:   rpi3_sdhost: unknown err, cmd = 0xd
ERROR:   rpi3_sdhost status: 0x10

Issues #17 and #33 are the same thing. As @rob-the-dude mentioned there, the following bit is key:

If I power-down the pi for a little while, the error USUALLY goes away and I can boot the next time. But a warm-reboot almost always hits it.

My observations:

• On cold boot it works, but if I let it idle on the UEFI setup for a couple of minutes it will mostly fail to boot afterwards.
• It definitely depends on SD card characteristics (a really old Transcend card works fine)
• Raspberry Pi firmware or TF-A is likely at fault, though as it's unclear which and the latter doesn't have a public issue tracker, hope it's okay to discuss it here.

@rob-the-dude: might be interesting to add what kind of SD card you're seeing this on, as I'm tallying 3x Samsung EVO thus far..

It can be reproduced with just ARM TF-A (I did use config.txt from UEFI release v1.36). Check out ARM TF-A:

CROSS_COMPILE=aarch64-unknown-linux-gnu- make PLAT=rpi3 \
PRELOADED_BL33_BASE=0x02000000 \
RPI3_PRELOADED_DTB_BASE=0x01000000 \
RPI3_DIRECT_LINUX_BOOT=1

I used UEFI release v1.36 as firmware base, then swapped out RPI_EFI.fd for just the TF-A build by modifying config.txt to boot build/rpi3/release/armstub8.bin:

armstub=armstub8.bin

I've been able to reproduce it on v2.3, v2.4, v2.5 of ARM TF-A; earlier releases fail to initialize the UART correctly. Someone on #17 suggests downgrading RPi3 UEFI to v1.24 worked for them; as that contains ARM TF-A v2.3 downgrading the Raspberry Pi firmware might offer additional clues...

from rpi3.

I had another go at hunting this down based on the clues provided in the linked issue - this time by bisecting the Raspberry Pi firmware:

11a76e07ef1b6304a378c4ee3da200fe6facea46 is the first bad commit
commit 11a76e07ef1b6304a378c4ee3da200fe6facea46
Author: popcornmix <[email protected]>
Date:   Thu Apr 16 12:40:43 2020 +0100

    firmware: Switch to building from common firmware branch

    firmware: isp: make AGC metering respect the (digital zoom) crop region

The previous commit (9e3c23ce77) boots fine whereas 11a76e07ef consistently fails to boot as long as it's a decently warm boot (left it to idle on UEFI setup for a few minutes prior to testing). This also means this project's release v1.23 is the last to ship a good firmware revision for SD cards affected.

Unfortunately, the commit message is rather nondescript, but does suggest an upstream branch switch is involved.

start.elf 11a76e07ef:

VC_BUILD_ID_USER: dom
VC_BUILD_ID_TIME: 12:33:18
VC_BUILD_ID_VARIANT: start
VC_BUILD_ID_TIME: Apr 16 2020
VC_BUILD_ID_BRANCH: bcm2711_2
VC_BUILD_ID_HOSTNAME: buildbot
VC_BUILD_ID_PLATFORM: raspberrypi_linux
VC_BUILD_ID_VERSION: 30355345c483638d3f8b013f58f0ee17094e7de2 (clean)

start.elf 9e3c23ce77:

VC_BUILD_ID_USER: dom
VC_BUILD_ID_TIME: 11:43:08
VC_BUILD_ID_BRANCH: master
VC_BUILD_ID_VARIANT: start
VC_BUILD_ID_TIME: Apr 15 2020
VC_BUILD_ID_HOSTNAME: buildbot
VC_BUILD_ID_PLATFORM: raspberrypi_linux
VC_BUILD_ID_VERSION: 82f9bb929ce2186eb1824178c1ae82902ad6275c (clean)

This doesn't rule out it ends up being a bug in TF-A, but someone from Raspberry Pi might be able to tell whether there are in fact relevant changes between these start.elf revisions.

Tagging in case they might be able to investigate on their end: @pelwell @popcornmix

from rpi3.

Just to warn you - that update contains probably the largest number of changes for non-Pi 4 devices.

from rpi3.

Well, I know it's been a LONG time since I chimed in on this, but I just encountered it again.

Same here - or well, I reboot my Raspberry Pi 3 infrequently enough to completely forget about needing a cold reboot and then end up Googling to re-read my own analysis..

I feel somewhat confident that the problem is in UEFI's handling of the RPI 3B+ SD interface.

More specifically that would be ARM TF-A (used by RPi UEFI). Refer my comment prior to this where I managed to reproduce it without UEFI and just TF-A.

I didn't dig any further in 2021, as proprietary VideoCore firmware is somehow involved which means it's best suited for the Raspberry Pi Foundation to sort out. Hardware documentation (for the RPi SDHOST controller) neither is available to the general public.

My best guess is that it's some sort of timing issue. The observation that the failure case can be avoided by cold booting (as in actually cold) is interesting - some sort of a clocking issue affected by temperature?

Things that might help:

• Comparing the way TF-A and u-boot configure/use the SDHOST controller. The drivers look pretty similar rpi3_sdhost.c bcm2835_sdhost.c
• A logic analyzer/scope trace of a good/bad SD on TF-A

Bonus, intriguing snippet from the BCM2835 ARM Peripherals document:

Additionally can the sampling clock for the response and data from the card be delayed in up
to 40 steps with a configurable delay between 200ps to 1100ps per step typically. The delay
is intended to cancel the internal delay inside the card (up to 14ns) when reading. The delay
per step will vary with temperature and supply voltage. Therefore it is better to use a bigger
delay than necessary as there is no restriction for the maximum delay.

Neither driver touches those registers, though.

from rpi3.

I spent some time comparing and debugging with both u-boot and TF-A. I've got a small patch that fixes this problem for me, though I have very limited ability to test it.

ARM-software/arm-trusted-firmware#1995

The basic idea is that some initial configuration parameters (clock rate, bus width) aren't configured into the hardware before commands start being sent. I suspect that the particular setting that matters is the "slow card" bit, but the initial clock setting also seemed wrong to me.

I'm certainly interested in feedback. I don't quite know how to push for this to be included in ARM's TF-A, but I've at least put it up here so folks can see it.

from rpi3.

The fix I provided has been merged into TF-A. A new build of UEFI for RPi3 with that fix included will hopefully resolve this issue.

from rpi3.

@rob-the-dude, thanks for doing this.

To keep in line with what we've been doing until now when it comes to TF-A, we'll have to wait for the formal v2.9 release so that we can use https://github.com/pftf/pitf to build the TF-A binaries.

Then, we'll need to submit these binaries for integration in https://github.com/tianocore/edk2-non-osi/tree/master/Platform/RaspberryPi/RPi3/TrustedFirmware.

And once that is done, we'll be able to build a new RPI3 UEFI firmware that integrates these changes...

from rpi3.

from rpi3.

this is confirmed to be a non-issue anymore ?

from rpi3.