PSCI HVC about qhypstub HOT 5 CLOSED

Hi @bryanodonoghue!

Implementing PSCI using HVC in EL2 should be indeed possible using a custom hyp firmware. I considered working on that at some point (mainly for a few MSM8916 devices that are unable to use the PSCI firmware from DB410c). There are two reasons why I have not worked on it so far:

1. My original motivation for working on qhypstub was to have EL2/virtualization available for the operating system (e.g. KVM on Linux or some other hypervisor). The PSCI implementation would block EL2, so you cannot use PSCI and virtualization at the same time.
2. There is of course quite some effort involved to have a properly working (and compliant) PSCI implementation that correctly configures the involved hardware components.

Since the hardware involved for SMP boot and CPU idle is pretty much identical to previous ARM32 SoCs like MSM8226 or MSM8974, I believe that many more platforms and users could benefit from finishing up the (non-PSCI) CPU idle related code in Linux instead. The cpuidle-qcom-spm driver is mainly lacking support for cluster idle (L2 SAW/SPM).

In my opinion the following setup is still quite clean and minimizes the maintenance overhead for the ARM64 Linux maintainers:

• SMP boot: boot with spin-table in bootloader (e.g. my implementation in lk2nd), officially supported
• CPU idle: use existing cpuidle-qcom-spm driver on ARM64, not officially supported (yet)

The CPU idle drivers are already abstracted by the cpuidle subsystem (just like many other driver subsystems). Unlike for SMP boot, there are no changes needed in ARM64 core code to make use of the cpuidle-qcom-spm driver. In fact, the only downstream change that I still have for SMP + cpuidle on ARM64 is this one (a simple fix to make the driver compile on ARM64): msm8916-mainline/linux@d0c1efe. Everything else is upstream and works automatically when using lk2nd (it detects if PSCI is supported and patches the DTB with spin-table and cpuidle-qcom-spm otherwise).

I expect that similar discussions will also come up for the Asahi Linux (Apple M1) project at some point. As far as I know the Apple M1 does not have EL3/PSCI and they almost certainly do want to use either their own hypervisor or KVM, without losing cpuidle. For SMP boot they also use spin-table already.

What do you think?

from qhypstub.

So I wasn't really aware of cpuidle-qcom-spm, to be honest I'm simultaneously surprised and not surprised at how little code there really is in that driver compared to downstream.

Certainly cluster support is missing, I see that from a quick look at (downstream) drivers/cpuidle/lpm-*.c

I like this answer though on cpuidle-qcom-spm if cluster support is all that's missing then agreed

• spin-table: aka lk2nd + DTB appendation
• cpuidle-qcom-spm + new code for cluster support

is the right way to do this with upstream.

Thanks !

from qhypstub.

So I wasn't really aware of cpuidle-qcom-spm, to be honest I'm simultaneously surprised and not surprised at how little code there really is in that driver compared to downstream.

Certainly cluster support is missing, I see that from a quick look at (downstream) drivers/cpuidle/lpm-*.c

I like this answer though on cpuidle-qcom-spm if cluster support is all that's missing then agreed

The code for the SPM moved a bit "back and forth" in mainline. It was originally at drivers/soc/qcom/spm.c (upstreamed by Lina Iyer at Linaro), then I moved it to cpuidle-qcom-spm.c to simplify usage on ARM64 and now it's split into cpuidle-qcom-spm and spm. I think in your case you first need to add the definitions for all the SPM/SAW blocks to the spm driver (on 8939: 8x CPU SAW, 2x L2 SAW, 1x CCI SAW). And cpuidle-qcom-spm would likely need to gain some CPU power domain support similar to cpuidle-psci-domain.c.

There are some old (work-in-progress) changes for L2 SAW support in the SPM driver that you could use as inspiration. Lina Iyer was working on that before work was shifted to the PSCI-related code (later taken over by Ulf Hansson as far as I understand). Some early draft commits used to be public on Lina Iyer's Linaro repository (somewhere at https://git.linaro.org/people/). It was deleted a few years ago when Lina Iyer left Linaro but I still have a backup of them. Let me know if you need them. They did help me to understand the basic concept a bit better, because the downstream code is extremely complex and confusing as you mentioned. :)

I have been meaning to work on this myself for quite some time but unfortunately I usually got stuck because of lack of time and all the complexity involved. Having this working would be essential for better power-saving on older Qualcomm SoCs supported by mainline Linux. :)

from qhypstub.

Thanks, this was a great discussion to follow.

For completeness (please call out if any of my understanding is wrong):

• In addition to losing the ability to run KVM/another hypervisor if we were to implement PSCI at EL2, it sounds like that solution wouldn't cover 100% of MSM8916/39 phones. Per the README, for secure boot-enabled devices lk2nd relies on a vulnerability that is only present in older firmware to inject qhypstub.
• The spin-table method would mean technically no hotplug support, but this is inconsequential given cpuidle. PSCI does distinguish between CPU stop (hotplug) and CPU suspend (cpuidle), but in practice the downstream stack (on both PSCI and non-PSCI platforms) never unplugs CPUs.
• For system suspend, platform_suspend_ops is not required as suspend2idle should be sufficient. PSCI doesn't make a distinction between system suspend and idling the last CPU, so we shouldn't need to either.
• We have a backlog of patches to add PM hooks in a bunch of places, those need to go upstream to unvote everything & get to lowest power state during system suspend (XO shutdown + Vdd minimization). Patches for MPM wakeup need to go in to be able to wake back up.

from qhypstub.

There is TF-A/ATF now as an open-source TZ firmware replacement that works also on MSM8939 (with partial CPU power management), at least on devices with firmware secure boot disabled. See https://trustedfirmware-a.readthedocs.io/en/latest/plat/qti-msm8916.html (upstream) and https://github.com/msm8916-mainline/arm-trusted-firmware (MSM8939 changes, not upstream yet).

For devices with secure boot I still think using spin-table and cpuidle-qcom-spm is the best approach. Older devices without virtualization support (e.g. MSM8974) would also benefit from extending the cpuidle-qcom-spm driver. Also, some people would like to use the virtualization functionality in Linux, which would not make EL2 available for implementing PSCI.

from qhypstub.