openmpdk / smdk Goto Github PK

I noticed that Samsung announced the latest 512GB CXL-DRAM. But there is no way to buy it. How do I contact? Thanks.

Because I do not have the CXL device and I have discovered that QEMU enables CXL 2.0 emulation. Therefore, I am interested in using SMDK with QEMU. Additionally, I came across a previous issue (#15) where errors were encountered while running SMDK on QEMU. I would like to inquire if there is a practical and reliable method to successfully run SMDK on QEMU.

Could you specify where "the Intelligent Tiering Engine" is implemented in SMDK?

What exactly does a single "CXL memory" refer to in the Wiki documents?(specifically, in section 2.3's Memory Partition)
Am I correct in saying that each "CXL.mem" block depicted at the hardware layer of "Zone Partition" or any "physical CXL memory" mentioned refers to a single memory expander device (e.g., the 512GB expander from this article: https://news.samsung.com/global/samsung-electronics-introduces-industrys-first-512gb-cxl-memory-module)

When I tested it using test_cxlswap.c, I found that I was being oom killed. Upon inspection, it seems that there is an issue with the avoid_oom function in util.h . you are passing the -17 parameter to oom_score_adj instead of oom_adj. But even after I made the modification, the program still cannot use the cxlswap interface properly; instead, it just stops directly when reaching the memory limit.

The network adapter link on my system became disabled (link down) after I booted into the SMDK kernels built with either SMDK 1.0 (linux-5.17-rc5-smdk) in Ubuntu 21.10 or SMDK 1.1 (linux-5.18.rc3-smdk) in Ubuntu 22.04. The network link would be normal and working if I booted into the generic kernels. What can I do to fix this issue?

Dear open-source contributor:
I encountered some issues while setting up the SMDK environment on my server. My environment consists of an Intel Xeon 4th Gen Intel Scalable Processor CPU(Sapphire Rapids ), a PCIe 5.0 compatible server, and Ubuntu 23.04 server edition with M88MX5891 controller. During the process of selecting the SMDK kernel through Grub, I encountered several problems. I will describe some of the issues below:

1.What settings are required to enable the SRAT table?Are there any other preconditions needed? How to set them?
2.When running ./dep_pkg_install.sh under SMDK/script, I encountered the following errors indicating that some libraries have been deprecated. How can I resolve this? What impact will it have if these libraries are missing? I am using the Alibaba source for package installation.http://mirrors.aliyun .com/ubuntu/
**E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
E: Package 'python' has no installation candidate
E: Package 'ctags' has no installation candidate
E: Package 'python-dev' has no installation candidate
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
E: Unable to locate package libjson-c3**

When I change back to the original official mirror source, I will give the following error:
**E: Unable to correct problems, you have held broken packages.
E: Unable to correct problems, you have held broken packages.
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
E: Unable to locate package libjson-c3**

3.After encountering the aforementioned issues, I still tried to proceed with the instructions provided at https://github.com/OpenMPDK/SMDK/wiki/3.-Installation.
Since it would be time-consuming to try on a physical host, I used Ubuntu 23.04 desktop in the virtual Box environment. I tried multiple versions of SMDK, and only version 1.2 allowed me to successfully select the 5.18.0-rc3-smdk kernel in Grub and boot into the Ubuntu system. However, there were other problems, such as the failure to build tools like py_smdk, bwd, cxl_cli, etc. Therefore, I would like to ask: Can the SMDK environment be tested in a virtual machine? It would be more convenient for me to test in a virtual machine instead of on the server host every time. I would appreciate a detailed build method and a comprehensive system environment (including software and hardware), which is not provided in detail at https://github.com/OpenMPDK/SMDK/wiki/3.-Installation.
Below is the log showing the failure to build cxl_cli.
**[build cxl_cli]

The Meson build system
Version: 1.0.1
Source dir: /home/hcm2304/Desktop/SMDK-smdk_v1.2.2/lib/cxl_cli
Build dir: /home/hcm2304/Desktop/SMDK-smdk_v1.2.2/lib/cxl_cli/build
Build type: native build
Project name: ndctl
Project version: 73
C compiler for the host machine: cc (gcc 12.2.0 "cc (Ubuntu 12.2.0-17ubuntu1) 12.2.0")
C linker for the host machine: cc ld.bfd 2.40
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wall: YES
Compiler for C supports arguments -Wchar-subscripts: YES
Compiler for C supports arguments -Wformat-security: YES
Compiler for C supports arguments -Wmissing-declarations: YES
Compiler for C supports arguments -Wmissing-prototypes: YES
Compiler for C supports arguments -Wnested-externs : NO
Compiler for C supports arguments -Wshadow: YES
Compiler for C supports arguments -Wsign-compare: YES
Compiler for C supports arguments -Wstrict-prototypes: YES
Compiler for C supports arguments -Wtype-limits: YES
Compiler for C supports arguments -Wmaybe-uninitialized: YES
Compiler for C supports arguments -Wdeclaration-after-statement: YES
Compiler for C supports arguments -Wunused-result: YES
Compiler for C supports arguments -D_FORTIFY_SOURCE=2: YES
Compiler for C supports arguments -O2: YES
meson.build:77: WARNING: Consider using the built-in warning_level option instead of using "-Wall".
meson.build:77: WARNING: Consider using the built-in optimization level instead of using "-O2".
Program git found: YES (/usr/bin/git)
Program env found: YES (/usr/bin/env)
WARNING: You should add the boolean check kwarg to the run_command call.
It currently defaults to false,
but it will default to true in future releases of meson.
See also: mesonbuild/meson#9300
Found pkg-config: /usr/bin/pkg-config (1.8.1)
Run-time dependency libkmod found: YES 30
Found CMake: /usr/bin/cmake (3.25.1)
Run-time dependency libudev found: NO (tried pkgconfig and cmake)

meson.build:139:0: ERROR: Dependency "libudev" not found, tried pkgconfig and cmake

A full log can be found at /home/hcm2304/Desktop/SMDK-smdk_v1.2.2/lib/cxl_cli/build/meson-logs/meson-log.txt
./build_lib.sh: line 246: cd: cxl_cli/build: No such file or directory

[build cxl_cli]..error**
4.Can I use the VirtualBox virtual machine to build an SMDK environment? Use only the"Test" provided to test system memory,Because virtual machine environments do not have CXL memory.

Hi, I' d like to ask which version of the CXL specification does SMDK support? Because wiki doesn' t mention supporting CXL 2.0 features such as switches and hot-plugging. Or SMDK doesn' t care about the version of CXL?
Also, I noticed that the SMDK includes Linux 6.0 kernel. Is it compatible with other Linux versions with CXL drivers?
Many thanks.

The SMDK looks quite promising and is great to have a real test. But due to some reasons, I don't have Samsung CXL memory Expander currently. I am wondering if the SMDK can support other CXL Memory Device including both Compatible and Optimization Paths

Many Thanks

Hi,

Nowadays, I'm glad to look at QEMU 7.2.50 supports the CXL2.0 emulation in:
https://www.qemu.org/docs/master/system/devices/cxl.html

And the latest dev branch for qemu cxl is enable to support cxl-type3 volatile memory:
https://gitlab.com/jic23/qemu/-/tree/cxl-2022-11-17

Now can SMDK tested/evaluated with QEMU now? If so, how to set it up?

Thanks in advance.

I built and installed the SMDK kernel 5.18-rc3-smdk under Ubuntu 22.04. I could see my CXL device in the lspci output and recognized as ExMem zone in the buddyinfo output as shown below.

But when I checked the CXL drivers, only cxl_core was loaded.

When I tried to load driver, cxl_pci.ko, I got below error.

Could you help decode the error messages?
If cxl_pci.ko is not loaded, how does it affect the CXL operation and testing?

The README for SMDK contains the following statement that refers to a figure "The SMDK is a collection of software tools and APIs that sit between applications and the hardware, as shown in the figure below." But the figure is missing.

Hi~~ In user space, we can use "s_malloc" to request CXL memory; so if I'm in kernel space, what interface should I use to request CXL memory?
Thanks a lot~~

hi,I am an employee of Samsung R&D Institute China-Xian. Recently, I need to use SMDK software in researching cxl related projects . Install & update the kernel without any problems.
Guide said if booting does not proceed and freezes with the "Loading Initial ramdisk" message after selecting the SMDK kernel, reboot using the kernel you previously used, enter the command below, and retry SMDK kernel booting.
$ cd /path/to/SMDK/lib/linux-5.18-rc3-smdk
$ sudo make modules_install -j 8
$ cd /lib/modules/{kernel_name}
$ sudo find . -name *.ko -exec strip --strip-unneeded {} +
$ cd /path/to/SMDK/lib/linux-5.18-rc3-smdk
$ sudo make install -j 8
$ sudo update-grub
After I did the above steps, the reboot still freezes at "Loading Initial ramdisk", is there any good solution, looking forward to your reply.

Server environment:
Linux smc-PowerEdge-R930 5.10.131 #1 SMP Tue Jul 19 18:15:52 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
smpboot: CPU0: Intel(R) Xeon(R) CPU E7-4809 v4 @ 2.10GHz (family: 0x6, model: 0x4f, stepping: 0x1)
total used free shared buff/cache available
Mem: 23988 1065 22309 23 613 22558
Swap: 2047 0 2047

Dear open-source contributor:
I encountered some issues while setting up the SMDK environment on my server. My environment consists of an Intel Xeon 4th Gen Intel Scalable Processor CPU(Sapphire Rapids ), a PCIe 5.0 compatible server, and Ubuntu 23.04 server edition with M88MX5891 controller. During the process of selecting the SMDK kernel through Grub, I encountered several problems. I will describe some of the issues below:

1.What settings are required to enable the SRAT table?Are there any other preconditions needed? How to set them?
2.When running ./dep_pkg_install.sh under SMDK/script, I encountered the following errors indicating that some libraries have been deprecated. How can I resolve this? What impact will it have if these libraries are missing? I am using the Alibaba source for package installation.http://mirrors.aliyun .com/ubuntu/
**E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
E: Package 'python' has no installation candidate
E: Package 'ctags' has no installation candidate
E: Package 'python-dev' has no installation candidate
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
E: Unable to locate package libjson-c3**

When I change back to the original official mirror source, I will give the following error:
**E: Unable to correct problems, you have held broken packages.
E: Unable to correct problems, you have held broken packages.
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
E: Unable to locate package libjson-c3**

3.After encountering the aforementioned issues, I still tried to proceed with the instructions provided at https://github.com/OpenMPDK/SMDK/wiki/3.-Installation.
 Since it would be time-consuming to try on a physical host, I used Ubuntu 23.04 desktop in the virtual Box environment. I tried multiple versions of SMDK, and only version 1.2 allowed me to successfully select the 5.18.0-rc3-smdk kernel in Grub and boot into the Ubuntu system. However, there were other problems, such as the failure to build tools like py_smdk, bwd, cxl_cli, etc. Therefore, I would like to ask: Can the SMDK environment be tested in a virtual machine? It would be more convenient for me to test in a virtual machine instead of on the server host every time. I would appreciate a detailed build method and a comprehensive system environment (including software and hardware), which is not provided in detail at https://github.com/OpenMPDK/SMDK/wiki/3.-Installation. 
Below is the log showing the failure to build cxl_cli.
**[build cxl_cli]

The Meson build system
Version: 1.0.1
Source dir: /home/hcm2304/Desktop/SMDK-smdk_v1.2.2/lib/cxl_cli
Build dir: /home/hcm2304/Desktop/SMDK-smdk_v1.2.2/lib/cxl_cli/build
Build type: native build
Project name: ndctl
Project version: 73
C compiler for the host machine: cc (gcc 12.2.0 "cc (Ubuntu 12.2.0-17ubuntu1) 12.2.0")
C linker for the host machine: cc ld.bfd 2.40
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wall: YES 
Compiler for C supports arguments -Wchar-subscripts: YES 
Compiler for C supports arguments -Wformat-security: YES 
Compiler for C supports arguments -Wmissing-declarations: YES 
Compiler for C supports arguments -Wmissing-prototypes: YES 
Compiler for C supports arguments -Wnested-externs : NO 
Compiler for C supports arguments -Wshadow: YES 
Compiler for C supports arguments -Wsign-compare: YES 
Compiler for C supports arguments -Wstrict-prototypes: YES 
Compiler for C supports arguments -Wtype-limits: YES 
Compiler for C supports arguments -Wmaybe-uninitialized: YES 
Compiler for C supports arguments -Wdeclaration-after-statement: YES 
Compiler for C supports arguments -Wunused-result: YES 
Compiler for C supports arguments -D_FORTIFY_SOURCE=2: YES 
Compiler for C supports arguments -O2: YES 
meson.build:77: WARNING: Consider using the built-in warning_level option instead of using "-Wall".
meson.build:77: WARNING: Consider using the built-in optimization level instead of using "-O2".
Program git found: YES (/usr/bin/git)
Program env found: YES (/usr/bin/env)
WARNING: You should add the boolean check kwarg to the run_command call.
         It currently defaults to false,
         but it will default to true in future releases of meson.
         See also: mesonbuild/meson#9300
Found pkg-config: /usr/bin/pkg-config (1.8.1)
Run-time dependency libkmod found: YES 30
Found CMake: /usr/bin/cmake (3.25.1)
Run-time dependency libudev found: NO (tried pkgconfig and cmake)

meson.build:139:0: ERROR: Dependency "libudev" not found, tried pkgconfig and cmake

A full log can be found at /home/hcm2304/Desktop/SMDK-smdk_v1.2.2/lib/cxl_cli/build/meson-logs/meson-log.txt
./build_lib.sh: line 246: cd: cxl_cli/build: No such file or directory

[build cxl_cli]..error**

The check in the below line is failing because of the change in __GFP_BITS_SHIFT definition.

Hi all,
do you have any timeframe for upstreaming patches in linux-5.18-rc3-smdk?

We did two different memcached experiments in a CXL enabled host booted using SMDK kernel. For both, we ran memcached server on CXL exposed memory.
First experiment, in default Zone partition setting, memcached server is run in exmem zone in compatible mode. Second experiment using numactl to redirect server traffic to CXL memory node1.
We tried to plot Latency vs TPS(Transactions Per Second), we observed better latency numbers with Zone partition setting compared to numactl test for Node-1 CXL memory.

Can you please let us know if this is expected ? and is there any optimization in memory allocation for Zone partition.
Following is the Configuration used:
export CXLMALLOC_CONF=use_exmem:true,exmem_zone_size:12288,normal_zone_size:12288,maxmemory_policy:oom,use_auto_arena_scaling:false,priority:exmem

Hi, First of all thank you for this opensource! 😄

How to slove the problem of [ 0.091860] ExMem empty ?
I appreciate it, if you help me! 😃

$ dmesg
...
[    0.088526] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
[    0.088542] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff]
[    0.088555] ACPI: SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff]
[    0.088707] NUMA: Initialized distance table, cnt=2
[    0.088738] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [m
em 0x00000000-0x107fffffff]
[    0.088865] NODE_DATA(0) allocated [mem 0x107ffc1000-0x107fffffff]
[    0.088996] NODE_DATA(1) allocated [mem 0x204ff9e000-0x204ffdcfff]
[    0.091805] Zone ranges:
[    0.091811]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.091831]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.091847]   Normal   [mem 0x0000000100000000-0x000000207fffffff]
[    0.091860]   ExMem    empty
[    0.091869]   Device   empty
[    0.091877] Movable zone start for each node
[    0.091891] Early memory node ranges
[    0.091896]   node   0: [mem 0x0000000000001000-0x000000000003dfff]
[    0.091910]   node   0: [mem 0x000000000003f000-0x000000000009ffff]
[    0.091919]   node   0: [mem 0x0000000000100000-0x000000006d90efff]
[    0.091929]   node   0: [mem 0x00000000777ff000-0x00000000777fffff]
[    0.091938]   node   0: [mem 0x0000000100000000-0x000000107fffffff]
[    0.091992]   node   1: [mem 0x0000001080000000-0x000000207fffffff]
[    0.092050] Initmem setup node 0 [mem 0x0000000000001000-0x000000107fffffff]
[    0.092074] Initmem setup node 1 [mem 0x0000001080000000-0x000000207fffffff]
[    0.092093] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.092110] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.092239] On node 0, zone DMA: 96 pages in unavailable ranges
[    0.103132] On node 0, zone DMA32: 40688 pages in unavailable ranges
[    0.245226] On node 0, zone Normal: 2048 pages in unavailable ranges
...
[    6.773184] cxlswap using pool (allocator: cxlbud)
[    6.773734] cxlcache using pool (allocator: cxlbud)
...
[    7.466205] cxl_pci 0000:3d:00.0: enabling device (0140 -> 0142)
[    7.469079] CXL: register cxl dvsec ranges
[    7.471867] CXL: node count: 0
[    7.474721] cxl_pci 0000:3d:00.0: No component registers (-19)
[    7.477450] cxl_pci 0000:3d:00.0: Failed to request region 0x0000000000001fff-0x000000000010201e
...
[   13.062696] cxl_mem mem0: CXL port topology not found
...

My system info:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

$ uname -r
6.4.0-smdk

$ cat /proc/buddyinfo 
Node 0, zone      DMA      2      2      1      0      0      0      0      0      1      1      2 
Node 0, zone    DMA32      6      6      6      5      4      4      3      5      6      2    413 
Node 0, zone   Normal    403     37     51     53     73     71     32      3     45     16  13110 
Node 1, zone   Normal     28   1896   1150    424    191     55      7      3      3      2  13893 

$ lsmod | grep cxl
cxl_mem                12288  0
cxl_port               16384  0
cxl_acpi               24576  0
cxl_pci               143360  1 cxl_acpi
cxl_core              229376  4 cxl_port,cxl_mem,cxl_pci,cxl_acpi

$ cd /lib/modules/6.4.0-smdk/kernel/drivers/cxl
$ tree
.
├── core
│   └── cxl_core.ko
├── cxl_acpi.ko
├── cxl_mem.ko
├── cxl_pci.ko
├── cxl_pmem.ko
└── cxl_port.ko

CXL Device info:

$ lspci -vvv
...
3d:00.0 CXL: Device 1b00:c000 (rev 01) (prog-if 10 [CXL Memory Device (CXL 2.x)])
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        NUMA node: 0
        Region 0: Memory at 20e000000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at 20c000000000 (64-bit, prefetchable) [size=128G]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Root Complex Integrated Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag- RBE+ FLReset+
                DevCtl: CorrErr- NonFatalErr- FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp+ 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
        Capabilities: [b0] MSI-X: Enable+ Count=7 Masked-
                Vector table: BAR=0 offset=00010000
                PBA: BAR=0 offset=00011000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC+ UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 8a-0a-f7-67-60-30-00-00
        Capabilities: [374 v1] Extended Capability ID 0x2f
        Capabilities: [3c0 v1] Designated Vendor-Specific: Vendor=1e98 ID=0000 Rev=1 Len=56: CXL
                CXLCap: Cache- IO+ Mem+ Mem HW Init+ HDMCount 1 Viral+
                CXLCtl: Cache- IO+ Mem- Cache SF Cov 0 Cache SF Gran 0 Cache Clean- Viral-
                CXLSta: Viral-
        Capabilities: [3f8 v1] Designated Vendor-Specific: Vendor=1e98 ID=000a Rev=0 Len=28 <?>
        Capabilities: [d04 v1] Designated Vendor-Specific: Vendor=1e98 ID=0008 Rev=0 Len=28 <?>
        Capabilities: [d48 v1] Extended Capability ID 0x2e
        Capabilities: [d60 v1] Designated Vendor-Specific: Vendor=8086 ID=0050 Rev=0 Len=12 <?>
        Capabilities: [d6c v1] Designated Vendor-Specific: Vendor=1e98 ID=0005 Rev=0 Len=16 <?>
        Capabilities: [d80 v1] Extended Capability ID 0x2e
        Kernel driver in use: cxl_pci
        Kernel modules: cxl_pci

안녕하세요,

src/container
위 디렉토리는 어디에 쓰는 건가요?

고맙습니다.

I tried following steps to build smdl_allocator code but facing issue in build "comp_api"

1. git clone https://github.com/OpenMPDK/SMDK.git
2. cd SMDK/lib/smdk_allocator
3. cd jemalloc-5.2.1/
4. ./autogen.sh && cd ..
5. make opt_api ///build done
6. make comp_api //build error

build error logs:
root@mukesh-Inspiron-7460:/home/smdk_allocator# make comp_api
cd jemalloc-5.2.1 && make -j
make[1]: Entering directory '/home/smdk_allocator/jemalloc-5.2.1'
make[1]: Leaving directory '/home/smdk_allocator/jemalloc-5.2.1'
mkdir -p lib
cd ./comp_api && make
make[1]: Entering directory '/home/smdk_allocator/comp_api'
g++ -Wall -shared -o ../lib/libcxlmalloc.so smdk_comp_api.o cxlmalloc.o cxlmalloc_cpp.o /home/smdk_allocator/comp_api/../core/init.o /home/smdk_allocator/comp_api/../core/alloc.o /home/smdk_allocator/comp_api/../core/config.o -L/home/smdk_allocator/comp_api/../jemalloc-5.2.1/lib/ -Wl,--whole-archive /home/smdk_allocator/comp_api/../jemalloc-5.2.1/lib/libjemalloc_pic.a -Wl,--no-whole-archive -lpthread -ldl -lnuma -lstdc++
/usr/bin/ld: /home/smdk_allocator/comp_api/../jemalloc-5.2.1/lib/libjemalloc_pic.a(jemalloc.pic.o): in function realloc': /home/smdk_allocator/jemalloc-5.2.1/src/jemalloc.c:2653: multiple definition of realloc'; smdk_comp_api.o:smdk_comp_api.c:(.text+0x12f): first defined here
/usr/bin/ld: /home/smdk_allocator/comp_api/../jemalloc-5.2.1/lib/libjemalloc_pic.a(jemalloc.pic.o): in function malloc': /home/smdk_allocator/jemalloc-5.2.1/src/jemalloc.c:2323: multiple definition of malloc'; smdk_comp_api.o:smdk_comp_api.c:(.text+0x20): first defined here
/usr/bin/ld: /home/smdk_allocator/comp_api/../jemalloc-5.2.1/lib/libjemalloc_pic.a(jemalloc.pic.o): in function posix_memalign': /home/smdk_allocator/jemalloc-5.2.1/src/jemalloc.c:2393: multiple definition of posix_memalign'; smdk_comp_api.o:smdk_comp_api.c:(.text+0x1ac): first defined here
.......
......
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:33: ../lib/libcxlmalloc.so] Error 1
make[1]: Leaving directory '/home/smdk_allocator/comp_api'
make: *** [Makefile:16: comp_api] Error 2

Dear SMDK contributors,

I am trying to build SMDK 2.0, but the build process got stuck while running header_install.sh with an error indicating that "CONFIG_EXMEM is leaked to user-space."

The error messages are as follows:

...
HDRINST usr/include/asm-generic/mman-common.h
error: include/uapi/asm-generic/mman-common.h: leak CONFIG_EXMEM to user-space
make[2]: *** [scripts/Makefile.headersinst:63: usr/include/asm-generic/mman-common.h] error 1
make[1]: *** [/home/user_name/SMDK/SMDK/lib/linux-6.6-smdk/Makefile:1298: headers] error 2
make: *** [Makefile:234: __sub-make] error 2

Here is my brief server information (CPU, memory, and kernel version):

[CPU] 2 * Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (3rd Gen)
[Board] Intel® Server Board M50CYP2SBSTD
[Memory] 8 * 64 GB @3200 MT/s DDR4 Synchronous LRDIMM Samsung M386A8K40DM2-CWE
[Linux kernel version] 6.6.0

Although our server does not support CXL and does not have CXL memory devices, I guess this error is orthogonal to the hardware specifications.
Additionally, building SMDK 1.5.1 did not produce this error, as there is no #ifdef CONFIG_EXMEM line in the mman-common.h header of SMDK 1.5.1.

I would like to ask if you have ever encountered this problem when building SMDK 2.0 and whether it can be resolved without modifying the kernel code.

Thank you!

Hi,

I'm currently trying to setup and use SMDK. As I don't have a CXL-capable CPU and a CXL device, I'm trying under a virtual machine using QEMU (w/ CXL modifications).

This repository includes QEMU, so I assumed that the SMDK was tested with QEMU. However, I was not able to get SMDK working under QEMU, even after forcefully creating the SRAT ACPI table under the virtual machine.
I noticed that the QEMU helper scripts under this repository don't include any CXL device declaration such as "-device cxl-type3".

Is SMDK tested/evaluated under QEMU?
If so, how can we set it up?
If not, what is the purpose of including the QEMU source code and the QEMU helper scripts?

Thanks in advance.

Location: https://github.com/OpenMPDK/SMDK/wiki/7.-Experiment-Results#741-methodology

Description: There is a minor typo in the description of "Cuse_bw" in the wiki. Attached is an image with a red circle indicating the error.

Incorrect Text: "Cuse_bw = in-use BW of DDR DRAM"

Corrected Text: "Cuse_bw = in-use BW of CXL RAM"

Attachment:

I'd like to know how SMDK provides a migration interface for userspace memory. And does SMDK have a kernel integration roadmap for CXLSwap, like using eBPF to inspect the dynamic usage of page cache and issue the CXLSwap?

Hi, I am looking at and studying the difference between Compatible and Optimized Paths and would like to analyze the Impact caused on the Same User Applications. As my understanding of Optimized Path from Wiki, user need to update the Memory Allocator API with New SMDK APIs, e.g. s_malloc，s_free. However, I have a quick check on s_malloc invocation and find the Memory Allocation Path seems s_malloc -> jemalloc -> Kernel System Call. So the Optimized Path still depends on the OS Kernel Space Efforts, Not a Totally User Space Work, so the Most efforts should be the Same with Compatible Path.

I thought before that the Optimized Path should be Useful for the Advanced Users who has CXL-aware features and can build up a Totaly User Space Memory Management Mechanism on top of CPU Host Memory Tiering, but it seems that is not the current CXL Optimized Direction.

Am I understanding Correctly?? It will be great if someone can help have a better understanding and on motivation and purpose of Optimized API Path.

Many Thanks

Hello, I noticed that there is a latency/bandwidth test step in the wiki. I do not have the CXL memory device at hand, but I am curious about the performance number of the CXL-memory product. Is it possible that you can share the results with me? Thank you!