Hi, I'm new to RPMA, and this concern is troubling me for days. In e

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Also, for Device DAX also doesn't work, same error here: <div class="snippet-clipb

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Got it, we have to change the memory lock size, check item 17 in <a href="https://www.

examples: RPMA file size limitation about rpma HOT 6 CLOSED

DanielLee343 commented on August 15, 2024

examples: RPMA file size limitation

from rpma.

Comments (6)

grom72 commented on August 15, 2024

Hi @DanielLee343 ,
can you attach logs?
It will be easier to follow the logic your example follows.

from rpma.

DanielLee343 commented on August 15, 2024

Hi @grom72,
Running the example 03 within 64K size of file is fine, but seems there's a maximum limit on the file size when calling ibv_reg_mr():

[lyuze@master 03-read-to-persistent]$ ./server 192.168.0.13 1234 /mnt/pmemfs0/file_larger_than_64k
memory size is: 65537
is pmem? 1
Jun 05 18:44:52.741986 [748654] *NOTE*  ep.c: 101: rpma_ep_listen: Waiting for incoming connection on 192.168.0.13:1234
Jun 05 18:44:52.742071 [748654] *ERROR* peer.c: 182: rpma_peer_mr_reg: ibv_reg_mr() failed: Cannot allocate memory

in which 65537 is 1 byte larger than 64K(65536)
Is there a proper solution if I want to perform RPMA R/W when file size is larger than 64K? Thanks!

from rpma.

DanielLee343 commented on August 15, 2024

Also, for Device DAX also doesn't work, same error here:

[lyuze@master 03-read-to-persistent]$ ./server 192.168.0.13 1234 /dev/dax1.0 
memory size is: 199229440
is pmem? 1
Jun 06 21:30:29.497260 [3243689] *NOTE*  ep.c: 101: rpma_ep_listen: Waiting for incoming connection on 192.168.0.13:1234
Jun 06 21:30:29.497342 [3243689] *ERROR* peer.c: 182: rpma_peer_mr_reg: ibv_reg_mr() failed: Cannot allocate memory
[lyuze@master 03-read-to-persistent]$ ndctl list -N
[
  {
    "dev":"namespace1.0",
    "mode":"devdax",
    "map":"dev",
    "size":799063146496,
    "uuid":"aa3d1d27-8fde-4258-b345-9cc97d3b4f0c",
    "chardev":"dax1.0",
    "align":2097152
  },
  {
    "dev":"namespace0.0",
    "mode":"fsdax",
    "map":"dev",
    "size":799063146496,
    "uuid":"3cb9cda2-bf7e-4012-a0a2-1e5fa2c13f79",
    "sector_size":512,
    "align":2097152,
    "blockdev":"pmem0"
  }
]

I guess there's some issue with my own configuration?

from rpma.

yangx-jy commented on August 15, 2024

@DanielLee343

Could you list all entries in /dev/infiniband/? like this:

# ll /dev/infiniband/
total 0
crw-rw-rw-. 1 root root  10, 122 Jun 10 14:01 rdma_cm
crw-rw-rw-. 1 root root 231, 192 Jun 10 14:01 uverbs0

from rpma.

DanielLee343 commented on August 15, 2024

@yangx-jy
I got this

[lyuze@master yuze]$ ll /dev/infiniband/
total 0
crw-rw-rw- 1 root root  10,  60 Jun  1 11:30 rdma_cm
crw-rw-rw- 1 root root 231, 192 Jun  1 10:23 uverbs0
crw-rw-rw- 1 root root 231, 193 Jun  1 10:23 uverbs1

from rpma.

DanielLee343 commented on August 15, 2024

Got it, we have to change the memory lock size, check item 17 in https://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-user

from rpma.

Recommend Projects