samsung / islet Goto Github PK

An on-device confidential computing framework

License: Apache License 2.0

Rust 88.28% Assembly 1.19% Python 4.71% Shell 2.32% Makefile 0.85% C++ 0.95% C 1.67% Dockerfile 0.03%

islet's Introduction

Islet is an open-source software project written in Rust that enables confidential computing on ARM architecture devices using the ARMv9 CCA. The primary objective of Islet is to enable on-device confidential computing and protect user privacy on end user devices.

While current confidential computing solutions mainly focus on server-side protection, it is equally important to safeguard user information at the user device level since that is where private data collection initially occurs. Furthermore, as more and more users rely on privacy apps such as private messengers, secure emails, password managers, and web browsers with privacy settings, there is a growing need to ensure privacy on user devices. Islet, an open-source project, addresses this need by providing a platform for ARM-based confidential computing.

Enabling CC on user devices will not only establish end-to-end CC throughout the entire data processing path, but it will also help create a secure computation model that enables processing of user private data on the user device using the same components that previously were employed at the server side without disclosing business logic. Furthermore, on-device confidential computing will be a key enabler for machine-to-machine computing without the need for server intervention

Feature Overview

Realm Management Monitor
Hardware Enforced Security
Confidential Computing API Standardization
Use case : Confidential Machine Learning

Overall Architecture

Islet provides a platform for running virtual machines (VMs) confidentially, with standard SDKs for easy integration with other confidential computing frameworks at upper layers. The platform consists of two key components: the Islet Realm Management Monitor (Islet-RMM) and Islet Hardware Enforced Security (Islet-HES).

Islet RMM operates at EL2 in the Realm world on the application processor cores and manages the confidential VMs, known as realms.
On the other hand, Islet HES performs device boot measurement, generates platform attestation reports, and manages sealing key functionality within a secure hardware IP apart from the main application processor.

In designing Islet, we aim to to address the current security challenges in confidential computing technologies right from the very beginning. To ensure that our software is built with safety in mind, we have chosen to use the Rust programming language, known for its unique security model that ensures memory safety and concurrency safety. Moving forward, we also plan to incorporate formal verification techniques to further enhance the security of our design and implementation.

For more information, please visit our developer site.

A demo video (Confidential ML)

This video shows how ISLET achieves an end-to-end confidential machine learning with a chat-bot scenario.
This video flows as follows.
1. It starts with a slide that describes all components involved in this demo. All components will run on confidential computing platforms.
2. (feed an ML model) The model provider feeds the ML model into the ML server. This is done through a secure channel established with the aid of the certifier framework.
3. (run a coding assistant) A mobile device user asks a chat-bot application that runs on ISLET for generating a function. And then, that request is passed on to the ML server through a secure channel. Finally, the user can see the result (i.e., function).
4. (launch a malicious server) This time, we launch a malicious server to show a failure case. When it attempts to join the certifier service (on the right side of the screen), it will not pass authentication as it results in a different measurement. Therefore, the malicious server cannot interact with the mobile device user in the first place.
To download this video, click here.

islet's People

Contributors

Stargazers

Watchers

Forkers

confidentialcomputing ysbnim holaymzhang caibirda fruitbox12 1proprogrammerchant zpzigi754 nook1208 l0czek icegrave0391 bokdeuk-jeong bitboom

islet's Issues

Fix CI failures by lack of runner's space

Fix a ci failure (sdk build)

The below is the related log.

(in islet/sdk)
$ cargo update
...
$ make simulated
   Compiling rsa v0.9.2
   Compiling rsi_el0 v0.0.1 (/home/runner/work/islet/islet/lib/rsi-el0)
error[E0432]: unresolved imports `nix::fcntl::OFlag`, `nix::sys::stat`
 --> lib/rsi-el0/src/lib.rs:6:11
  |
6 | use nix::{fcntl::OFlag, libc::O_RDWR, sys::stat::Mode};
  |           ^^^^^^^^^^^^                     ^^^^ could not find `stat` in `sys`
  |           |
  |           no `OFlag` in `fcntl`

error[E0433]: failed to resolve: could not find `ioctl_read` in `nix`
  --> lib/rsi-el0/src/kernel.rs:13:10
   |
13 |     nix::ioctl_read!(abi_version, b'x', 190u8, u32);
   |          ^^^^^^^^^^ could not find `ioctl_read` in `nix`

error[E0433]: failed to resolve: could not find `ioctl_readwrite_buf` in `nix`
  --> lib/rsi-el0/src/kernel.rs:14:10
   |
14 |     nix::ioctl_readwrite_buf!(measurement_read, b'x', 192u8, RsiMeasurement);
   |          ^^^^^^^^^^^^^^^^^^^ could not find `ioctl_readwrite_buf` in `nix`

error[E0433]: failed to resolve: could not find `ioctl_write_buf` in `nix`
  --> lib/rsi-el0/src/kernel.rs:15:10
   |
15 |     nix::ioctl_write_buf!(measurement_extend, b'x', 193u8, RsiMeasurement);
   |          ^^^^^^^^^^^^^^^ could not find `ioctl_write_buf` in `nix`
...

Refactor PageTable interfaces

Move "entires: [E; N]" from core PageTable interfaces to each implementations.

Reinforce granule management

Currently, it uses the big lock for the granule table. We have to turn this into per-entry lock in accordance with RMM spec.
Any other security checks additionally needed?? --> figure it out too.

Replace ManuallyDrop with a more idiomatic Rust way

Java benchmark tool 선정

attestation 지원을 위한 AMD SEV 코드 분석

KVM/Qemu내 AMD SEV용으로 attestation 지원을 위한 코드들 분석

RMI status

FID	RMM Spec	Islet supported
0xC4000150	RMI_VERSION	O
0xC4000151	RMI_GRANULE_DELEGATE	O
0xC4000152	RMI_GRANULE_UNDELEGATE	O
0xC4000153	RMI_DATA_CREATE	X
0xC4000154	RMI_DATA_CREATE_UNKNOWN	X
0xC4000155	RMI_DATA_DESTROY	X
0xC4000157	RMI_REALM_ACTIVATE	X
0xC4000158	RMI_REALM_CREATE	O
0xC4000159	RMI_REALM_DESTROY	X
0xC400015A	RMI_REC_CREATE	X
0xC400015B	RMI_REC_DESTROY	X
0xC400015C	RMI_REC_ENTER	X
0xC400015D	RMI_RTT_CREATE	X
0xC400015E	RMI_RTT_DESTROY	X
0xC400015F	RMI_RTT_MAP_UNPROTECTED	X
0xC4000161	RMI_RTT_READ_ENTRY	X
0xC4000162	RMI_RTT_UNMAP_UNPROTECTED	X
0xC4000164	RMI_PSCI_COMPLETE	X
0xC4000165	RMI_FEATURES	O
0xC4000166	RMI_RTT_FOLD	X
0xC4000167	RMI_REC_AUX_COUNT	O
0xC4000168	RMI_RTT_INIT_RIPAS	X
0xC4000169	RMI_RTT_SET_RIPAS	X

old-fashioned rust toolchian issue (including llvm issue)

Issue 1

When i tried to apply cargo bloat to CI, our old-fashioned toolchain makes error.

Issue 2

So, i bumped rust to v1.60.0 using LLVM 14.0.0 and i replaced llvm_asm! to asm!.
But that version has below issues.

asm!("smc #0x0"); // error: instruction requires: el3
asm!("msr vttbr_el2, x3"); // error: expected writable system register or pstate

ref: llvm-issue

A safe interface to read/write page table

Currently, we write a page table entry in a typical way, (e.g., self.0 = RawPTE::new(0);), but this is not a safe way to modify page table entries, as it requires a strict order of instructions.
- e.g., secure way: str something to pte; barrier_instructions; tlb_instructions; it should be read/written atomically and proper barrier or tlb instructions must be followed right after.
As of now, this is not a big problem as we don't modify RTT after realm kernel booting. But, once it gets implemented, the current way to access page_table might cause subtle bugs.

Move "FVP binary" away from islet-asset

it might be that redistributing FVP binaries in such a way is prohibited.

Sporadic realm linux boot failure

Symptom

No boot message from realm linux

Related code

Linux realm got stuck while processing fdt in setup_arch():

ffff800008c1387c <setup_arch>:
....
ffff800008c13924:   94000e7b    bl  ffff800008c17310 <fixmap_remap_fdt>
ffff800008c13928:   aa0003f3    mov x19, x0
ffff800008c1392c:   b50000e0    cbnz    x0, ffff800008c13948 <setup_arch+0xcc> 
// <=== result of fixmap_remap_fdt() is often not zero. 
ffff800008c13930:   aa1303e2    mov x2, x19
ffff800008c13934:   9101c3e1    add x1, sp, #0x70
ffff800008c13938:   d0fff820    adrp    x0, ffff800008b19000 <kallsyms_token_index+0x1250>
ffff800008c1393c:   911de000    add x0, x0, #0x778
ffff800008c13940:   97f42b11    bl  ffff80000891e584 <_printk>
ffff800008c13944:   14000009    b   ffff800008c13968 <setup_arch+0xec>
ffff800008c13948:   b9806fe1    ldrsw   x1, [sp, #108]
ffff800008c1394c:   f9403be0    ldr x0, [sp, #112]
ffff800008c13950:   97d8c0fc    bl  ffff800008243d40 <memblock_reserve>
ffff800008c13954:   aa1303e0    mov x0, x19
ffff800008c13958:   9400cbfc    bl  ffff800008c46948 <early_init_dt_scan>
ffff800008c1395c:   72001c1f    tst w0, #0xff
ffff800008c13960:   54000081    b.ne    ffff800008c13970 <setup_arch+0xf4>  // b.any
ffff800008c13964:   17fffff3    b   ffff800008c13930 <setup_arch+0xb4>
ffff800008c13968:   d503203f    yield    // Realm is stuck here
ffff800008c1396c:   17ffffff    b   ffff800008c13968 <setup_arch+0xec>

Corresponding linux source code:

static void __init setup_machine_fdt(phys_addr_t dt_phys)
{
    int size;
    void *dt_virt = fixmap_remap_fdt(dt_phys, &size, PAGE_KERNEL);
    const char *name;

    if (dt_virt)
        memblock_reserve(dt_phys, size);

    if (!dt_virt || !early_init_dt_scan(dt_virt)) {  // failed
        pr_crit("\n"
            "Error: invalid device tree blob at physical address %pa (virtual address 0x%px)\n"
            "The dtb must be 8-byte aligned and must not exceed 2 MB in size\n"
            "\nPlease check your bootloader.",
            &dt_phys, dt_virt);

        /*
         * Note that in this _really_ early stage we cannot even BUG()
         * or oops, so the least terrible thing to do is cpu_relax(),
         * or else we could end-up printing non-initialized data, etc.
         */
        while (true)
            cpu_relax();  // <===== here we stuck at this point: corresponds to the 'yield' above

Workaround

Improper cache operation or tlb invalidation seems to cause this. Configure FVP to have a single core.

diff --git a/scripts/.config b/scripts/.config
index 512365cf8956..fde2734733c4 100644
--- a/scripts/.config
+++ b/scripts/.config
@@ -14,7 +14,7 @@ bp.secure_memory=1
 bp.smsc_91c111.enabled=1
 bp.ve_sysregs.exit_on_shutdown=1
 cache_state_modelled=0
-cluster0.NUM_CORES=4
+cluster0.NUM_CORES=1
 cluster0.PA_SIZE=48
 cluster0.check_memory_attributes=0
 cluster0.cpu0.DCZID-log2-block-size=4
@@ -36,7 +36,7 @@ cluster0.restriction_on_speculative_execution=2
 cluster0.restriction_on_speculative_execution_aarch32=2
 cluster0.rme_support_level=2
 cluster0.stage12_tlb_size=1024
-cluster1.NUM_CORES=4
+cluster1.NUM_CORES=0
 cluster1.PA_SIZE=48
 cluster1.check_memory_attributes=0
 cluster1.cpu0.DCZID-log2-block-size=4

Host와 Guest OS간에 제공되는 메모리 공유 기법 파악하기

virtio 제외하고,
기존에 있는 Host (KVM, Kernel 또는 QEMU)와 메모리 공유하는 기법이 있는지 파악하기

Update assets for ACS

ACS currently does not use the islet's assets. Thus, it would be common for users to face with build errors with the following command: ./scripts/fvp-cca --normal-world=acs --rmm=islet. For example, cmake should be updated to a version over 3.19 and srec_cat should be installed.

Linux Realm SMP error

[Need to modify KVM to properly handle psci cpu-on requests sent from the realm linux]
How to reproduce: run qemu with -smp 2(-smp 4 will cause another error. use -smp 2)

Sequence of Realm Linux kernel to boot secondary CPUs

smp_init()  kernel/smp.c
    --> bringup_nonboot_cpus(setup_max_cpus);  kernel/cpu.c
        --> cpu_up(cpu, CPUHP_ONLINE);   kernel/cpu.c
            --> _cpu_up() kernel/cpu.c
                -->  boot_secondary() ./arch/arm64/kernel/smp.c
                    --> ops = get_cpu_ops(cpu);   ./arch/arm64/kernel/smp.c
                         ops->cpu_boot(cpu);
                         --> cpu_psci_cpu_boot() ./arch/arm64/kernel/psci.c 
                              (cpu_psci_ops .cpu_boot = cpu_psci_cpu_boot)
                            --> psci_ops.cpu_on(cpu, pa_secondary_entry) drivers/firmware/psci/psci.c
                                --> hvc call 로 normal world의 kvm에 psci 처리 요청

HVC call handler in KVM for the request from the realm linux

        handle_exit()   arch/arm64/kvm/handle_exit.c
            --> handle_hvc()   arch/arm64/kvm/handle_exit.c
                --> kvm_hvc_call_handler()   arch/arm64/kvm/hypercalls.c
                    --> kvm_psci_call(vcpu); --> kvm_psci_0_2_call()   arch/arm64/kvm/psci.c
                        --> kvm_psci_vcpu_on()    arch/arm64/kvm/psci.c
                            --> cpu_id = smccc_get_arg1(source_vcpu);
                            --> reset_state->pc = smccc_get_arg2(source_vcpu);
                            --> reset_state->r0 = smccc_get_arg3(source_vcpu);
                            --> kvm_make_request(KVM_REQ_VCPU_RESET, vcpu);
                                --> kvm_reset_vcpu() arch/arm64/kvm/reset.c
                                    --> *vcpu_pc(vcpu) = target_pc;   // rmi call로 전달안함
                                    --> vcpu_set_reg(vcpu, 0, reset_state.r0);  // r0만 rmi call로 SetReg

vcpu_set_reg() used for updating most of vcpu registers is not used to update pc in the secondary vCPUs on the PSCI CPU-ON request. As we implanted a SetReg RMI request in the vcpu_set_reg() API, update request for the pc of secondary VCPUs is not captured.
*vcpu_pc(vcpu) = target_pc;

Add vcpu_set_reg(vcpu, 31, target_pc) where '*vcpu_pc(vcpu) = target_pc; ' is called.

https://github.com/Samsung/islet-linux/commit/04b710f9aaecb0a70cffb670c07ddbe85943224b

Fix the CI failure

Currently, islet-ci / linux-cca does not succeed.

The related log is like the below.

In Build linux cca components:
...
[!] Building SDK...
[!] Failed to run: ['make', 'fvp'] @ /home/runner/work/islet/islet/sdk/
...
thread '<unnamed>' panicked at 'assertion failed: pos.get() <= self.position()', compiler/rustc_metadata/src/rmeta/encoder.rs:426:9
error: failed to run custom build command for `islet_sdk v0.1.0 (/home/runner/work/islet/islet/sdk)`

Caused by:
  process didn't exit successfully: `/home/runner/work/islet/islet/out/release/build/islet_sdk-52085967a6e1bdc0/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at 'Unable to generate bindings.: CargoMetadata("/home/runner/work/islet/islet/sdk/Cargo.toml", Metadata(Output { status: ExitStatus(unix_wait_status(25856)), stdout: "", stderr: " Downloading crates ...\n  Downloaded winapi-util v0.1.5\n  Downloaded hermit-abi v0.3.2\n  Downloaded errno v0.3.1\n  Downloaded instant v0.1.12\n  Downloaded windows-targets v0.48.1\n  Downloaded redox_syscall v0.3.5\n  Downloaded errno-dragonfly v0.1.2\n  Downloaded hermit-abi v0.1.19\n  Downloaded wasi v0.11.0+wasi-snapshot-preview1\n  Downloaded linked_list_allocator v0.10.5\n  Downloaded spin v0.9.8\n  Downloaded spinning_top v0.2.5\n  Downloaded cc v1.0.79\n  Downloaded windows_aarch64_gnullvm v0.48.0\n  Downloaded windows_x86_64_gnullvm v0.48.0\n  Downloaded windows_i686_msvc v0.48.0\n  Downloaded windows_x86_64_msvc v0.48.0\n  Downloaded windows_aarch64_msvc v0.48.0\n  Downloaded windows_x86_64_gnu v0.48.0\n  Downloaded windows_i686_gnu v0.48.0\nerror: failed to unpack package `windows_i686_gnu v0.48.0`\n\nCaused by:\n  failed to unpack entry at `windows_i686_gnu-0.48.0/lib/libwindows.0.48.0.a`\n\nCaused by:\n  failed to unpack `/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/windows_i686_gnu-0.48.0/lib/libwindows.0.48.0.a`\n\nCaused by:\n  failed to unpack `windows_i686_gnu-0.48.0/lib/libwindows.0.48.0.a` into `/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/windows_i686_gnu-0.48.0/lib/libwindows.0.48.0.a`\n\nCaused by:\n  No space left on device (os error 28)\n" }))', sdk/build.rs:36:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: could not compile `minicbor`
make: *** [Makefile:16: sdk] Error 101

Error: Process completed with exit code 1.

Manage the memory region state

Granule structure

ref. tf-rmm

Granule state

pub enum GranuleState {
    Undelegated,
    Delegated,
    RD,
    Rec,
    RecAux,
    Data,
    RTT,
}

data structure

pub struct Granule {
    state: GranuleState,
    refcount: usize,
    idx: usize,
}

Realm is unexpectedly destroyed

The below is the related log from the latest commit (a95bf0f).
Not sure whether it is the cause of the bus error or the result of the bus error.

...
[TRACE]monitor::event -- RMI: RTT_READ_ENTRY       [88160E000, 1FFE00000, 2] > [0, 2, 0, 0, 0]
[TRACE]monitor::event -- RMI: REC_DESTROY          [881BDA000, 80000000, 801BDA000] > [0]
[TRACE]monitor::event -- RMI: GRANULE_UNDELEGATE   [881BDA000] > [0]
[TRACE]monitor::event -- RMI: GRANULE_UNDELEGATE   [881787000] > [0]
...
[TRACE]monitor::event -- RMI: REALM_DESTROY        [88160E000] > [0]
[TRACE]monitor::event -- RMI: GRANULE_UNDELEGATE   [88160E000] > [0]
[TRACE]monitor::event -- RMI: GRANULE_UNDELEGATE   [882C98000] > [0]
...

Enable ACS with islet-rmm

Support wasm runtime as a realm

Task

Select a wasm runtime for realm el0
Select kernel for realm for realm el1
Compose wasm realm

Conclusion

I made realm-wasm with wamer + linux, by embedding a wasmer into initrd.
This is the simplest way to make realm-wasm through statically built into rootfs for linux.

The reason why i didn't choose unikraft + wasm runtime combo is that there is no combo fully supporting aarch64 without modification.

Below is a summary of data i investigated.

Summary

kernel	runtime	aarch64	x86_64	emulator	remarks
linux	wasmer	O	O	qemu-user-aarch64	latest version supported (3.1.4)
linux	wasmtime	O	O	qemu-user-aarch64	aarch64: stable-0.26.1 only supported
unikraft	wamr	X	O	n/a
unikraft	rust	X	O	n/a
unikraft	c	O	O	qemu-system-aarch64
nanos	c	O	O	qemu-system-aarch64
nanos	wasmer	X	O	qemu-system-aarch64
nanos	wasmedge	X	O	qemu-system-aarch64

Java App 내 secret code 구분과 실행 방법 모색

SEV, TDX에서는 virtio를 어떻게 지원하는지 파악

IPA State

RIPAS

현재 RMM 코드 파악

RMM 직접 설정 레지스터
islet/rmm/armv9a/src/helper/mod.rs
KVM이 설정한 값 그대로 RMM이 사용중인 레지스터
아래 커밋 참조 (trusted-firmware-a)
commit 7d9e25f2e40990a63c7dc45d22ad026a4296be04 (HEAD)
Date: Mon Jul 4 18:30:07 2022 +0900
[temporal] context: do not save/restore some sysregs
RMM이 KVM에 전달해서 설정하는 레지스터

값 설정하는 곳: islet/rmm/armv9a/src/exception/trap.rs
handle_lower_exception() 내 tf.regs 값 설정부 참고
KVM으로 전달하는 곳: islet/rmm/armv9a/src/rmi/ream.rs
rmi::Code::VMRun listener 코드

        trace!("Switched to VCPU {} on VM {}", vcpu, vm);
        let ret = rmm_exit();

        match ret[0] {
            rmi::RET_SUCCESS => call.reply(rmi::RET_SUCCESS),
            rmi::RET_EXCEPTION_TRAP | rmi::RET_EXCEPTION_IRQ => {
                call.reply(ret[0]).or(Err("RMM failed to reply."))?;
                call.reply(ret[1]);
                call.reply(ret[2]);
                call.reply(ret[3])
            }

KVM에서 받은 값 설정하는 곳
islet/linux/arch/arm64/kvm/hyp/vhe/switch.c
__kvm_vcpu_run_vhe() 함수 참고

위의 방법은 초기에 정의한 RMI 인터페이스를 활용해서 linux 실행이 되게 하는 것에 초점을 맞춘 것으로,
'전혀' 좋은 선택은 아닙니다. 염두에 두시고 변경 방향 고려해 주세요~

RMM debug 메시지 출력 방법
islet/rmm/board/fvp/src/entry.rs
diff --git a/rmm/board/fvp/src/entry.rs b/rmm/board/fvp/src/entry.rs
index 95553aa..277a6db 100644
--- a/rmm/board/fvp/src/entry.rs
+++ b/rmm/board/fvp/src/entry.rs
@@ -54,7 +54,7 @@ unsafe fn clear_bss() {

 fn init_console() {
     let _ = stdout().attach(crate::driver::uart::pl011::device());
-    logger::register_global_logger(LevelFilter::Info); // Control log level
+    logger::register_global_logger(LevelFilter::Debug); // Control log level
     info!("Initialized the console!");
 }

virtio-blk-device enable하기

(virtio-blk-device는 virtio-mmio와 vring에 기반해서 virtio-blk 디바이스가 동작한다. virtual pci에 기반한 virtio-blk-pci과는 bus가 다르다. 동작의 단순화를 위해 virtio-blk-pci 대신 virtio-blk-device로 먼저 리눅스 realm을 대상으로 시도해 봄)
현재 realm VM 실행하면서 매핑하는 메모리는 모두 realm PAS에 매핑한다. (모니터를 통해 GPT 업데이트를 한다.)
realm VM이 NW와 공유해야하는 메모리 예)

gic의 command buffer
virtio-mmio page
virtio-OOO device들을 virtqueue
그런데 이 메모리 영역들이 어디 인지 알 수 없다.

[ naive한 방법 1]
685b279

어느 IPA가 realm과 공유해야하는 메모리인지 모르니까, realm이 액세스하는 모든 메모리를 최대한 다 NW와 공유하도록 설정한다.
(i.e, S2 page table에 매핑할 때 pte에 NS field를 1로 세팅하고, GPT는 업데이트 하지 않고 그대로 둔다. 부팅할 때 이미 NW 소유로 GPT에 설정돼 있어서)

결과: linux realm 부팅 단계에서 vda device를 잘 인식하고, fdkisk로 파티션 생성, mkfs.ext2 (ramdisk이미지에 ext2밖에 없다)로 fs초기화도 성공.
단 mount하면서 에러가 발생
mount /dev/vda1 ttt
[ 8207.601339] Unable to handle kernel unknown 40 at virtual address 0000007f78ac0400
[ 8207.601529] Mem abort info:
[ 8207.601619] ESR = 0x96000068
[ 8207.601719] EC = 0x25: DABT (current EL), IL = 32 bits
[ 8207.601869] SET = 0, FnV = 0
[ 8207.601977] EA = 0, S1PTW = 0
[ 8207.602079] FSC = 0x28: unknown 40 <== GPT fault 발생
[ 8207.602189] Data abort info:
[ 8207.602289] ISV = 0, ISS = 0x00000068
[ 8207.602399] CM = 0, WnR = 1
[ 8207.602614] user pgtable: 4k pages, 39-bit VAs, pgdp=00000008803b6000
[ 8207.602780] [0000007f78ac0400] pgd=0800000883dad003, p4d=0800000883dad003, pud=0800000883dad003, pmd=0800000880841003, pte=00e800088700ef43
[ 8207.603169] Internal error: Oops: 96000068 [#1] PREEMPT SMP
[ 8207.603322] Modules linked in:
[ 8207.603429] CPU: 4 PID: 538 Comm: qemu-system-aar Not tainted 5.17.0-g826588446686-dirty #17
[ 8207.603659] Hardware name: FVP Base RevC (DT)
[ 8207.603779] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 8207.603989] pc : __arch_copy_to_user+0x30/0x240
[ 8207.604160] lr : _copy_to_iter+0x100/0x654
[ 생략]
[ 8207.606919] Call trace:
[ 8207.607010] __arch_copy_to_user+0x30/0x240
[ 8207.607169] p9_client_read_once+0x100/0x234
[ 8207.607334] p9_client_read+0x68/0x1cc
[ 8207.607479] v9fs_file_read_iter+0x88/0xa0
[ 8207.607659] new_sync_read+0xd4/0x150
[ 8207.607827] vfs_read+0x190/0x1dc
[ 8207.607991] ksys_pread64+0x78/0xc0
[ 8207.608155] __arm64_sys_pread64+0x24/0x30
[ 8207.608329] invoke_syscall+0x48/0x114
[ 8207.608499] el0_svc_common.constprop.0+0x44/0xec
[ 8207.608669] do_el0_svc+0x24/0x84
[ 8207.608829] el0_svc+0x2c/0x84
[ 8207.608982] el0t_64_sync_handler+0x1a4/0x1b0
[ 8207.609161] el0t_64_sync+0x18c/0x190
[ 8207.609339] Code: 54000240 cb040042 36000084 38401423 (380008c3)
[ 8207.609499] ---[ end trace 0000000000000000 ]---

todo: host(NW)에 있는 qemu 프로세스에서 에러가 발생했다. realm으로 설정된 메모리 중에 nw와 share 되어야 하는게 더 있나보다.

Fix a bus fault in realm booting

The bus faults occurred during realm linux's booting, because threads (tx/rx/ctrl) spawned from kvmtool access a realm's region for virtio's communication. Note that kvmtool allocates and mmaps the region for the realm, and the threads shares the same address space with kvmtool. A possible reason for the bus fault would be that we have not configured the I/O region as the shared region (I will examine this later).

The triggering point in nw-linux.

(case 1) # Timeline 2-0

ffffffc0089131a4 <__pi_clear_page>:    [nw-linux]
ffffffc0089131a4:	d503245f 	bti	c
ffffffc0089131a8:	d53b00e1 	mrs	x1, dczid_el0
ffffffc0089131ac:	37200121 	tbnz	w1, #4, ffffffc0089131d0 <__pi_clear_page+0x2c>
ffffffc0089131b0:	12000c21 	and	w1, w1, #0xf
ffffffc0089131b4:	d2800082 	mov	x2, #0x4                   	// #4
ffffffc0089131b8:	9ac12041 	lsl	x1, x2, x1
ffffffc0089131bc:	d50b7420 	dc	zva, x0              <- the point of crash

[  102.798894] Call trace:
[  102.799005]  clear_page+0x18/0x4c
[  102.799171]  get_page_from_freelist+0x458/0xbbc
[  102.799369]  __alloc_pages+0xd4/0xbe0
[  102.799550]  __folio_alloc+0x14/0x20
[  102.799730]  vma_alloc_folio+0x94/0x1dc
[  102.799918]  alloc_zeroed_user_highpage_movable+0x3c/0x50
[  102.800142]  __handle_mm_fault+0xfd0/0x18c0
[  102.800324]  handle_mm_fault+0xf4/0x1b0
[  102.800496]  do_page_fault+0x17c/0x400
[  102.800686]  do_mem_abort+0x44/0x94
[  102.800879]  el0_da+0x30/0x90
[  102.801066]  el0t_64_sync_handler+0xf8/0x140
[  102.801282]  el0t_64_sync+0x190/0x194

(case 2) # Timeline 2-1

ffffffc0089134b0 <__pi_copy_page>:
...
ffffffc0089134e4:	91020021 	add	x1, x1, #0x80
ffffffc0089134e8:	f2402c1f 	tst	x0, #0xfff
ffffffc0089134ec:	d503201f 	nop
ffffffc0089134f0:	a8300c02 	stnp	x2, x3, [x0, #-256]   <- the point of crash

[  204.064152] Call trace:
[  204.064267]  copy_page+0x40/0xd0
[  204.064435]  copy_user_highpage+0x18/0x40
[  204.064636]  __handle_mm_fault+0x940/0x18c0
[  204.064821]  handle_mm_fault+0xf4/0x1b0
[  204.064995]  do_page_fault+0x17c/0x400
[  204.065191]  do_mem_abort+0x44/0x94
[  204.065379]  el0_da+0x30/0x90
[  204.065572]  el0t_64_sync_handler+0xf8/0x140
[  204.065792]  el0t_64_sync+0x190/0x194

The triggering point in linux realm (while setting virtual queue during virtio's virtnet probing) # Timeline 0
https://github.com/Samsung/islet-asset/blob/c865d47546eeaa45fae3a4e2e3a689b8f7502a72/drivers/virtio/virtio_mmio.c#L446

in drivers/virtio/virtio_mmio.c
static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int index,
				  void (*callback)(struct virtqueue *vq),
				  const char *name, bool ctx) {
    ...
    writel(1, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);   // this line indirectly causes a fault (mmio write)
    ...
}

The triggering point in kvmtool (in handling the above VIRTIO_MMIO_QUEUE_READY) # Timeline 1
https://github.com/Samsung/islet-asset/blob/2e8fe7ff61243527c6360b6e463b75d1bd8db408/include/kvm/virtio.h#L169

in ./include/kvm/virtio.h
static inline bool virt_queue__available(struct virt_queue *vq)
    ...
    vring_avail_event(&vq->vring) = last_avail_idx;  // this line indirectly causes a fault
    ...

Timeline flows from Timeline 0 -> Timeline 1 -> Timeline 2-0 or Timeline 2-1.

Do optimizations in frequently called RMIs

Some RMI commands (e.g., REC_ENTER) are called lots of times so might be critical to overall performance. Make such jobs as fast as possible. (e.g., a more efficient data structure)

VM 실행시 GPT 매핑 에러

증상:

RMM 메시지:
[WARN]armv9a::rmi::realm -- failed to set GPT 886B6B000

Monitor 메시지:
ERROR: RMMD: PAS Transition failed. GPT ret = -22, PA: 0x886b6b000, FID = 0xc40001b0

내 PC에서 실행시켰을 때는 정상 동작함. 새로 repo를 새로 다운로드 받아 증상 재현 확인함. 기존 로컬 버전과의 차이점 파악

Figure out a better way to associate Rec with Rd

Currently,

pub struct Rec {
    pub rd: &'static Rd,
    vcpuid: usize,
    ripas: Ripas,
}

One Rd can have multiple Recs. So, roughly thinking, we need to have the ability to manage the reference count of the rd. (need to look at RMM spec and TF-RMM, and figure out what would be the best way to associate Rec with Rd)

EL3 monitor쪽의 Realm attestation 관련 지원 사항 조사

https://github.com/ARM-software/arm-trusted-firmware

Get granule state according to a given PA

...

So, we have to check the addr is DELEGATED state before calling smc. I let this work for next PR (undelegation TCs).

...

Originally posted by @bitboom in #135 (comment)

Reinforce RMI input/context validations

The goal of this issue is to implement all validation checks in the RMM spec for RMI commands.

e.g., according to the RMM spec,

B3.3.12 RMI_REC_CREATE command
...
B3.3.12.2 Failure conditions
ID --- condition
params_align --- conditions
...
rd_state --- conditions

Unikernel Porting

Candidate (KVM supported)

Osv

https://github.com/cloudius-systems/osv

MirageOS

https://github.com/mirage/mirage

rusty-hermit (rust)

https://github.com/hermitcore/rusty-hermit

[Cache] Garbage value in shared region (NS, Realm)

I found that RMM reads dummy values (0xCFDFDFDF) with ACS tests.

Read failed with ACS: Params 0x88306000
Read succeed with TFTF: Params 0x88C01000

Originally posted by @bitboom in #145 (comment)

Ask for Suggestions about Research Topics

Hi, Thanks for your great work.

I am an AI researcher from Swinburne University of Technology in Australia. I watched your code prediction demo and want to conduct research related to ISLET. May I ask if there are any particular topics you care about, so maybe my team could contribute to your work.

Thanks again.

ACS(Architecture Compliance Suite) Status

Status

REGRESSION REPORT: 
==================
   TOTAL TESTS     : 51 (8 skipped)
   TOTAL PASSED    : 30
   TOTAL FAILED    : 18
   TOTAL SKIPPED   : 2
   TOTAL SIM ERROR : 1

Details

no.	category	testcase	Passed	Partially Passed	Failed	Assignee
1	interface	cmd_granule_delegate	O			#135
2	interface	cmd_granule_undelegate	O			#148
3	interface	cmd_realm_create	O			#155
4	interface	cmd_realm_activate	O			#183
5	interface	cmd_rtt_read_entry	O			#156, #193, #201
6	interface	cmd_rtt_init_ripas	O			#193, #205
7	interface	cmd_rtt_create	O			#193, #205
8	interface	cmd_data_create	O			#183, #193, #202
9	interface	cmd_data_create_unknown	O			#193, #212
10	interface	cmd_rtt_map_unprotected	O			#220, #156
11	interface	cmd_rec_create		O (35/36)		#185
12	interface	cmd_rec_enter	O			#198
13	interface	cmd_rmi_version	O			#154
14	interface	cmd_rsi_version	O			#154
15	interface	cmd_realm_config		O		#154
16	interface	cmd_ipa_state_get	O			#154
17	interface	cmd_ipa_state_set			O	@jinbpark
18	interface	cmd_multithread_realm_up			O
19	interface	cmd_multithread_realm_mp			O
20	interface	cmd_secure_test			O
21	memory management	mm_ripas_change	O
22	memory management	mm_ripas_change_reject	O
23	memory management	mm_ripas_change_partial	O
24	memory management	mm_hipas_assigned_ripas_empty_da			O	@zpzigi754
25	memory management	mm_hipas_destroyed_ripas_ram_da			O
26	memory management	mm_hipas_assigned_ripas_empty_ia			O
27	memory management	mm_hipas_assigned_ia			O
28	memory management	mm_hipas_assigned_da	O			#197
29	memory management	mm_unprotected_ipa_boundary	O			#197
30	memory management	mm_protected_ipa_boundary			O	@nook1208
31	memory management	mm_gpf_exception			O
32	memory management	mm_rtt_translation_table	O			#197
33	memory management	mm_rtt_fold_assigned			O
34	memory management	mm_rtt_fold_unassigned			O
35	memory management	mm_rtt_fold_u_ipa_assigned			O
36	memory management	mm_rtt_fold_destroyed			O
37	memory management	mm_hipas_destroyed_ripas_ram_ia			O
38	memory management	mm_hipas_unassigned_ripas_empty_da			O
39	memory management	mm_hipas_unassigned_ripas_empty_ia			O
40	memory management	mm_hipas_destroyed_ripas_empty_da			O
41	memory management	mm_hipas_destroyed_ripas_empty_ia			O
42	memory management	mm_hipas_unassigned_ripas_ram_da			O
43	memory management	mm_hipas_unassigned_ripas_ram_ia			O
44	memory management	mm_feat_s2fwb_check_1	O			#197
45	memory management	mm_rtt_level_start	O			#193
46	exception	exception_rec_exit_wfe			O
47	exception	exception_rec_exit_wfi			O
48	exception	exception_rec_exit_hostcall	O			#156
49	exception	exception_rec_exit_psci			O
50	exception	exception_realm_unsupported_smc	O			#156
51	exception	exception_rec_exit_hvc	O			#156, #203
52	exception	exception_rec_exit_ripas	O			#197
53	exception	exception_rec_exit_ia	O			#197
54	exception	exception_emulatable_da			O
55	exception	exception_non_emulatable_da			O
56	exception	exception_non_emulatable_da_1			O
57	exception	exception_non_emulatable_da_2			O
58	gic	gic_hcr_invalid			O
59	gic	gic_ctrl_list_invalid	O			#198

Related Issue

Realm linux kernel booting log (stuck point)

This is the latest log about the stuck point. I leave it for the reference.

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x00000000]
[    0.000000] Linux version 6.2.0-rc1-gc865d47546ee-dirty (changho@changho) (aarch64-none-linux-gnu-gcc (Arm GNU Toolchain 11.3.Rel1) 11.3.1 20220712, GNU ld (Arm GNU Toolchain 11.3.Rel1) 2.38.20220708) #49 SMP PREEMPT Thu Jul  6 18:42:23 KST 2023
[    0.000000] Machine model: linux,dummy-virt
[    0.000000] RME: RSI version 1.0 advertised
[    0.000000] earlycon: pl11 at MMIO 0x000000001c0a0000 (options '')
[    0.000000] printk: bootconsole [pl11] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] NUMA: No NUMA configuration found
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] NUMA: NODE_DATA [mem 0x8ff73a00-0x8ff75fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000008fffffff]
[    0.000000] cma: Reserved 32 MiB at 0x000000008c400000
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.2
[    0.000000] percpu: Embedded 20 pages/cpu s42728 r8192 d31000 u81920
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Hardware dirty bit management
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Fallback order for Node 0: 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 64512
[    0.000000] Policy zone: DMA
[    0.000000] Kernel command line:  console=ttyS0 root=/dev/vda rw  console=pl011,mmio,0x1c0a0000 console=ttyAMA0 printk.devkmsg=on
[    0.000000] random: crng init done
[    0.000000] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.000000] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 1.
[    0.000000] software IO TLB: mapped [mem 0x0000000088400000-0x000000008c400000] (64MB)
[    0.000000] Memory: 121156K/262144K available (9472K kernel code, 962K rwdata, 2732K rodata, 1600K init, 482K bss, 108220K reserved, 32768K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=1.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: 32 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x000000003ffd0000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 timer(s) running at 100.00MHz (virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0x1ffffffffffffff max_cycles: 0x171024e7e0, max_idle_ns: 440795205315 ns
[    0.000004] sched_clock: 57 bits at 100MHz, resolution 10ns, wraps every 4398046511100ns
[    0.007598] Console: colour dummy device 80x25
[    0.008730] Calibrating delay loop (skipped), value calculated using timer frequency.. 200.00 BogoMIPS (lpj=400000)
[    0.008863] pid_max: default: 32768 minimum: 301
[    0.009295] LSM: initializing lsm=capability,integrity
[    0.011317] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.012200] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.014753] Spurious percpu IRQ12 on CPU0
[    0.070603] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    0.104326] cblist_init_generic: Setting adjustable number of callback queues.
[    0.104408] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.107874] cblist_init_generic: Setting shift to 0 and lim to 1.
[    0.120313] rcu: Hierarchical SRCU implementation.
[    0.120375] rcu: 	Max phase no-delay instances is 1000.
[    0.130800] EFI services will not be available.
[    0.141180] smp: Bringing up secondary CPUs ...
[    0.141238] smp: Brought up 1 node, 1 CPU
[    0.141317] SMP: Total of 1 processors activated.
[    0.141385] CPU features: detected: Branch Target Identification
[    0.141466] CPU features: detected: ARMv8.4 Translation Table Level
[    0.141542] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    0.141637] CPU features: detected: Common not Private translations
[    0.141716] CPU features: detected: CRC32 instructions
[    0.141786] CPU features: detected: Data independent timing control (DIT)
[    0.141868] CPU features: detected: E0PD
[    0.141930] CPU features: detected: Enhanced Counter Virtualization
[    0.142020] CPU features: detected: RCpc load-acquire (LDAPR)
[    0.142095] CPU features: detected: LSE atomic instructions
[    0.142168] CPU features: detected: Privileged Access Never
[    0.142242] CPU features: detected: RAS Extension Support
[    0.142314] CPU features: detected: Random Number Generator
[    0.142388] CPU features: detected: Speculation barrier (SB)
[    0.142460] CPU features: detected: Stage-2 Force Write-Back
[    0.142537] CPU features: detected: TLB range maintenance instructions
[    0.143210] CPU: All CPU(s) started at EL1
[    0.143294] alternatives: applying system-wide alternatives
[    0.189495] devtmpfs: initialized
[    0.222596] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.222728] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.237759] DMI not present or invalid.
[    0.249876] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.292676] DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
[    0.323034] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    0.353423] DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.355272] audit: initializing netlink subsys (disabled)
[    0.368726] audit: type=2000 audit(0.000:1): state=initialized audit_enabled=0 res=1
[    0.368838] cpuidle: using governor menu
[    0.370915] hw-breakpoint: found 2 breakpoint and 2 watchpoint registers.
[    0.371050] ASID allocator initialised with 65536 entries
[    0.382105] Serial: AMBA PL011 UART driver
[    0.396964] KASLR enabled
[    0.414556] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages
[    0.414639] HugeTLB: 16380 KiB vmemmap can be freed for a 1.00 GiB page
[    0.414735] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages
[    0.414817] HugeTLB: 508 KiB vmemmap can be freed for a 32.0 MiB page
[    0.414902] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages
[    0.414994] HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page
[    0.415088] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages
[    0.415170] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page
[    0.452616] iommu: Default domain type: Translated 
[    0.452687] iommu: DMA domain TLB invalidation policy: strict mode 
[    0.459546] SCSI subsystem initialized
[    0.467988] pps_core: LinuxPPS API ver. 1 registered
[    0.468056] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <[email protected]>
[    0.468214] PTP clock support registered
[    0.486697] clocksource: Switched to clocksource arch_sys_counter
[    0.518650] VFS: Disk quotas dquot_6.6.0
[    0.519647] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.620214] NET: Registered PF_INET protocol family
[    0.627724] IP idents hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.636044] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.636980] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.637122] TCP established hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.640546] TCP bind hash table entries: 2048 (order: 4, 65536 bytes, linear)
[    0.653370] TCP: Hash tables configured (established 2048 bind 2048)
[    0.667503] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.669222] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.671176] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.691989] RPC: Registered named UNIX socket transport module.
[    0.692062] RPC: Registered udp transport module.
[    0.692127] RPC: Registered tcp transport module.
[    0.692188] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.702813] kvm [1]: HYP mode not available
[    0.710786] Initialise system trusted keyrings
[    0.718649] Unpacking initramfs...
[    1.024726] ------------[ cut here ]------------
[    1.024785] Unexpected interrupt (irqnr 75)
[    1.025160] WARNING: CPU: 0 PID: 9 at drivers/irqchip/irq-gic-v3.c:697 gic_handle_irq+0x118/0x124
[    1.025305] Modules linked in:
[    1.025375] CPU: 0 PID: 9 Comm: kworker/u2:0 Not tainted 6.2.0-rc1-gc865d47546ee-dirty #49
[    1.025500] Hardware name: linux,dummy-virt (DT)
[    1.025572] Workqueue: events_unbound async_run_entry_fn
[    1.025710] pstate: 614000c5 (nZCv daIF +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[    1.025832] pc : gic_handle_irq+0x118/0x124
[    1.025925] lr : gic_handle_irq+0x118/0x124
[    1.026017] sp : ffff800008003fe0
[    1.026079] x29: ffff800008003fe0 x28: ffff000001068000 x27: 0000000000000d30
[    1.026253] x26: 0000000000000000 x25: ffffd7c5d41ac000 x24: ffff8000080937b8
[    1.026427] x23: 0000000081400005 x22: ffffd7c5d35ec488 x21: ffff800008093730
[    1.026602] x20: 000000000000004b x19: 000000000000004b x18: ffffffffffffffff
[    1.026773] x17: ffff283a3bea5000 x16: ffff800008004000 x15: ffff800088003c57
[    1.026950] x14: 0000000000000000 x13: ffffd7c5d41bff98 x12: 00000000000004f2
[    1.027120] x11: 00000000000001a6 x10: ffffd7c5d4217f98 x9 : ffffd7c5d41bff98
[    1.027294] x8 : 00000000ffffefff x7 : ffffd7c5d4217f98 x6 : 0000000000000000
[    1.027464] x5 : 000000000000bff4 x4 : 0000000000000000 x3 : 0000000000000000
[    1.027630] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000001068000
[    1.027798] Call trace:
[    1.027848]  gic_handle_irq+0x118/0x124
[    1.027942]  call_on_irq_stack+0x2c/0x5c
[    1.028046]  do_interrupt_handler+0x80/0x84
[    1.028164]  el1_interrupt+0x34/0x70
[    1.028277]  el1h_64_irq_handler+0x18/0x2c
[    1.028398]  el1h_64_irq+0x64/0x68
[    1.028489]  release_pages+0x74/0x470
[    1.028614]  folio_batch_move_lru+0x100/0x1cc
[    1.028743]  folio_add_lru+0x7c/0xe0
[    1.028869]  filemap_add_folio+0x60/0xc4
[    1.028967]  __filemap_get_folio+0x210/0x410
[    1.029072]  pagecache_get_page+0x18/0x70
[    1.029194]  grab_cache_page_write_begin+0x18/0x24
[    1.029323]  simple_write_begin+0x2c/0xf4
[    1.029443]  generic_perform_write+0xa8/0x1e0
[    1.029578]  __generic_file_write_iter+0x11c/0x1a4
[    1.029688]  generic_file_write_iter+0x78/0xfc
[    1.029792]  __kernel_write_iter+0x94/0x200
[    1.029928]  kernel_write+0xd0/0x190
[    1.030018]  xwrite.constprop.0+0x4c/0xc4
[    1.030140]  do_copy+0xc8/0x124
[    1.030255]  write_buffer+0x38/0x58
[    1.030368]  flush_buffer+0x4c/0xbc
[    1.030487]  __gunzip+0x27c/0x30c
[    1.030573]  gunzip+0x1c/0x28
[    1.030661]  unpack_to_rootfs+0x154/0x278
[    1.030782]  do_populate_rootfs+0x68/0x12c
[    1.030906]  async_run_entry_fn+0x34/0xe0
[    1.031028]  process_one_work+0x1d0/0x320
[    1.031141]  worker_thread+0x14c/0x444
[    1.031247]  kthread+0x10c/0x110
[    1.031339]  ret_from_fork+0x10/0x20
[    1.031440] ---[ end trace 0000000000000000 ]---

Linux VM 내 커널에서 Host OS와 메모리 공유하는 부분 파악

Guest code 내 ITS command queue로 사용할 메모리 영역을 설정 하는 부분

its_probe_one() (guest-linux/drivers/irqchip/irq-gic-v3-its.c)
  --> page = alloc_pages_node(....)
  --> its->cmd_base = (void *)page_address(page);
   --> baser = (virt_to_phys(its->cmd_base) | 각종 flag들
   --> gits_write_cbaser(baser, its->base + GITS_CBASER)

Guest code 내 ITS command queue로 command 삽입 부 (예)

its_send_single_command() 
-->rd_idx = readl_relaxed(its->base + GITS_CREADR); 
-->next_cmd = its_post_commands(its);
      --> writel_relaxed(wr, its->base + GITS_CWRITER); // NW와 메모리 공유가 되지 않으면 KVM에서 이것 처리하다가 패닉 발생

KVM에서 ITS command 처리부

KVM의 request처리

static void vgic_its_process_commands(struct kvm *kvm, struct vgic_its *its)
{
    gpa_t cbaser; 
   u64 cmd_buf[4];

    /* Commands are only processed when the ITS is enabled. */
    if (!its->enabled)
        return;

    cbaser = GITS_CBASER_ADDRESS(its->cbaser);

    while (its->cwriter != its->creadr) {
        int ret = kvm_read_guest_lock(kvm, cbaser + its->creadr,
                          cmd_buf, ITS_CMD_SIZE);

KVM과 Realm간 메모리 공유가 되지 않았을 때 KVM의 에러 스택 trace

[ 3147.185710]  __arch_copy_from_user+0x94/0x230
[ 3147.185882]  kvm_read_guest+0x6c/0xa4
[ 3147.186035]  vgic_its_process_commands.part.0+0xa8/0x970
[ 3147.186220]  vgic_mmio_write_its_cwriter+0x80/0xa4
[ 3147.186390]  dispatch_mmio_write+0xd0/0x130
[ 3147.186570]  __kvm_io_bus_write+0xb4/0xf0
[ 3147.186732]  kvm_io_bus_write+0x48/0x80
[ 3147.186880]  io_mem_abort+0x1dc/0x220
[ 3147.187040]  kvm_handle_guest_abort+0x1f0/0xa64
[ 3147.187210]  handle_exit+0x70/0x180
[ 3147.187370]  kvm_arch_vcpu_ioctl_run+0x1cc/0x5e4
[ 3147.187530]  kvm_vcpu_ioctl+0x23c/0x950

RMM에서 현재 workaround로 처리해 놓은 부분

islet/rmm/armv9/src/rmi/realm.rs

commit cf24cfa
Author: Bokdeuk Jeong [email protected]
Date: Mon Jul 4 18:23:54 2022 +0900

[temporal/rmi]:VMMapMemory - set gic its cbaser address shared

Set NS bit for a specific ipa to share the page
between normal world(kvm) and realm (kernel).
This is a temporary solution until we design how to share memory between them.

이 커밋을 제거하면 위의 KVM stack trace상의 에러가 발생합니다.

linux-cca changes

check the linux-cca changes

compare changes
listing to-do
prepare the commit for nw-linux

Run tf-rmm on fvp

Work flow

Bump tf-a to v2.8-lts
Bump tf-a-tests to v2.8-lts
Import tf-rmm-v0.2.0
Run trp w/ tf-a & tf-a-tests v2.8 (1 failed: multi cpu delegation)
- Debugging
Run tf-rmm w/ tf-a & tf-a-tests v2.8 (1 hang: multi cpu delegation)
- Debugging
Bump fvp to latest
Run islet-rmm w/ tf-a & tf-a-tests v2.8

[TF-A] [LTS v2.8.0] RC0 tag has been created
Hello, As announced in the last Tech forum, the code freeze date for the LTS branch was Feb 3, 2023.

References

TF-A refers to a non-existent commit

enable memory management unit

make memory region on linker file (memory.x)
- .xlat_tables
make page translation table
- static ref MMU_PAGE
register setting
changes:
- MAIR_EL2, TCR_EL2 (ref. tf-rmm)
- set HCR_EL2.E2H to 1
- set SCTLR_EL2.M to 1

RMM refactor

refactor armv9a codes

Items

change call procedure regarding as VM(i.e event_handler)
move and separate routines not related with armv9a to monitor
consider to expand other handlers

적용할 타겟 Java app 선정

Tips and note

Toolchain

FVP

Opensource activation

Below jobs are working in progress at rfc branch

Restore commit history

Issue: History of upstream are squash-ed even though they are third-party
Solution: Restore history
linux
guest-linux
islet-build
islet

Manage third-party projects

Issue
- Redundant forked repos (islet-build, islet-linux)
- Almost sub-modules needs to be modified (tf-a, tf-a-tests, build)
Solution: Use worktree which manages multiple working trees

Before

~/islet (main) $ tree -L 1
. 
├── assets # submodule
├── build # submodule, forked, modified
├── guest-linux # submodule
├── linux # submodule, forked, modified
├── trusted-firmware-a # submodule, modified
└── upstream-trusted-firmware-a # submodule

After

~/islet (rfc) $ tree -L 2
.
├── assets # submodule
├── third-party
│   ├── nw-linux # worktree
│   ├── optee-build # worktree
│   ├── realm-linux # worktree
│   ├── tf-a # worktree
│   └── tf-a-tests # worktree

Third-Party Projects

usage	project	modified	branch (worktree)	upstream base
hypervisor	normal world linux	O (kvm)	nw-linux-5.17	upstream-linux-v5.17
realm	realm linux	X	realm-linux	upstream-linux-v5.19
monitor	tf-a	O	tf-a	upstream-tf-a-v2.7
el-2, realm	tf-a-tests	O	tf-a-tests	upstream-tf-a-tests-v2.7)
building linux	optee-build	O	optee-build	6a868e3

how to get access to nw-linux again

Hi, I want to know how to get access to normal world linux again after "./launch.sh", using tf-rmm and fvp
#42 mentioned that it's impossible to exit from realm to nw-linux
But I just want to get access to the control of nw-linux again
so is there any method to close the realm
I'm not familiar with the using of fvp, so I'm not sure if it can be simply done by fvp, Maybe like lauching realm in a new terminal so terminal for nw-linux can be used to close the realm

Figure out a way to securely manage "Run" and "HostMMIO_Read/Write"

Problem-1: mmio read/write

We leave the task of read/write to Host and wait for Host to complete it.
But it's hard to say this is secure at all times. We need to figure out a way to securely handle this.

Problme-2: gprs in Run

There may be a possibility that some private information in Realm is leaked through gprs in Run.
We need to figure out a way to securely handle this.

struct _Inner {
    imm: u16,
    gprs: [u64; HOST_CALL_NR_GPRS],  // does this info leak some information to Host??
}

NOTE: overall, it feels like similar to IAGO attacks in terms of how to exploit these two things.

About Launch Realm in FVP

Fix SMC Context bug

Below line should be fixed to set to x0, which mean arg0.

            match self.on_event.get(&ctx.cmd) {
                Some(handler) => {
                    let res = ctx.do_rmi(|arg, ret| handler(arg, ret, monitor));
                    if let Err(val) = res {
                        ctx.set_ret0(val.into()); << arg0
                    } else {
                        ctx.set_ret0(rmi::SUCCESS); << arg0
                    }
                }
                None => {
                    error!("Not registered event: {:X}", ctx.cmd);
                    ctx.init_arg(&[rmi::RET_FAIL]); << like this
                }