memfd-based allocator exhausts on more than 20 peers about rosenpass HOT 18 OPEN

I was able to reproduce this issue on my setup (Ubuntu 23.10, Rust 1.80 nightly, apracar:generate_config branch, both DUT and ATE on same machine). To set RLIMIT_MEMLOCK, I ran ulimit -l <size> in a shell before executing rosenpass exchange-config.

In case it helps, for 50 peers, I found by trial and error that the minimum <size> that doesn't cause allocation failure is about 32000, which (since the unit of <size> is 1024-byte blocks, see man bash) comes out to about 32MB of locked memory required.

For 100 peers, it takes about 61MB of locked memory, and for 10 peers, it takes about 7MB, so as a rule of thumb it looks like ~700kB/peer is sufficient.

Can I point you at the following line of code:

pub type SPk = Secret<{ StaticKem::PK_LEN }>; // Just Secret<> instead of Public<> so it gets allocated on the heap

This is the classic mc eliece public key, which is hundreds of KB. Its the single biggest key we have and its a public key. This problem will probably become drastically less severe if we use a Box here or a heap allocated version of Public< ... > for the McEliece key specifically.

from rosenpass.

I think we could estimate the amount of secret memory needed per peer and then just do a back of the envelope estimation within the application to check if there will be enough memory…

We could also change the limit from within the application.

Finally we might need a privileged startup handler at some point anyway and that can certainly set the limit.

I think there also is some overly generous use of secret memory in a couple places in the code. We could optimize here, actually treating secret memory as a precious resource.

from rosenpass.

I was able to reproduce this issue on my setup (Ubuntu 23.10, Rust 1.80 nightly, apracar:generate_config branch, both DUT and ATE on same machine). To set RLIMIT_MEMLOCK, I ran ulimit -l <size> in a shell before executing rosenpass exchange-config.

In case it helps, for 50 peers, I found by trial and error that the minimum <size> that doesn't cause allocation failure is about 32000, which (since the unit of <size> is 1024-byte blocks, see man bash) comes out to about 32MB of locked memory required.

For 100 peers, it takes about 61MB of locked memory, and for 10 peers, it takes about 7MB, so as a rule of thumb it looks like ~700kB/peer is sufficient.

from rosenpass.

@pqcfox Would it be possible for you to retry with the latest in #347 to see how this improvement fairs against your previous test?

from rosenpass.

I just verified that this indeed works! I set ulimit -l 2048 for the DUT (smaller than that of the Raspberry Pi) and exchange-config for 50 peers worked without a hitch. I'll toss a review on the PR right now!

Also, love the idea of moving the McEliece pubkey to the heap--@Prabpreet, would you want to include that in your PR, or would it be easier for me to toss up a separate tiny PR for that and then rebase once you merge yours?

from rosenpass.

Note to reproducer/fixer:

Memory that can be allocated is limited by rlimit parameters when using memfd_secret- a soft limit by the process, and a hard limit by a privileged process.

https://man7.org/linux/man-pages/man2/getrlimit.2.html

from rosenpass.

Note to reproducer/fixer:

Memory that can be allocated is limited by rlimit parameters when using memfd_secret- a soft limit by the process, and a hard limit by a privileged process.

https://man7.org/linux/man-pages/man2/getrlimit.2.html

Did we integrate the memfd-secret feature as an optional feature or is it enabled by default?

Why do different limits apply for memfd secret here? Is it due to the maximum allowed number of file descriptors?

from rosenpass.

Did we integrate the memfd-secret feature as an optional feature or is it enabled by default?

Currently it is enabled by default. I'll put it behind a feature flag for the time being till we figure out this issue.

I did not anticipate this limit to be so low- we should definitely have a peer exhaustion test- there is likely an upper limit for malloc based allocation too.

Why do different limits apply for memfd secret here? Is it due to the maximum allowed number of file descriptors?

From my understanding, the limit that applies is RLIMIT_MEMLOCK. In the conventional malloc strategy, the guard pages are not under mlock protection, rather are guarded by mprotect. Unfortunately, any space we allocate under memfd_secret is protected with mlock here including guard pages, so we hit the memlock limit quicker.

          This is the maximum number of bytes of memory that may be
          locked into RAM.  This limit is in effect rounded down to
          the nearest multiple of the system page size.  This limit
          affects [mlock(2)](https://man7.org/linux/man-pages/man2/mlock.2.html), [mlockall(2)](https://man7.org/linux/man-pages/man2/mlockall.2.html), and the [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) MAP_LOCKED
          operation.  Since Linux 2.6.9, it also affects the
          [shmctl(2)](https://man7.org/linux/man-pages/man2/shmctl.2.html) SHM_LOCK operation, where it sets a maximum on
          the total bytes in shared memory segments (see [shmget(2)](https://man7.org/linux/man-pages/man2/shmget.2.html))
          that may be locked by the real user ID of the calling
          process.  The [shmctl(2)](https://man7.org/linux/man-pages/man2/shmctl.2.html) SHM_LOCK locks are accounted for
          separately from the per-process memory locks established
          by [mlock(2)](https://man7.org/linux/man-pages/man2/mlock.2.html), [mlockall(2)](https://man7.org/linux/man-pages/man2/mlockall.2.html), and [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html) MAP_LOCKED; a
          process can lock bytes up to this limit in each of these
          two categories.

          Before Linux 2.6.9, this limit controlled the amount of
          memory that could be locked by a privileged process.
          Since Linux 2.6.9, no limits are placed on the amount of
          memory that a privileged process may lock, and this limit
          instead governs the amount of memory that an unprivileged
          process may lock.

One fix can be to attempt to increase the limits in the allocator when we start hitting these limits by a certain amount.

from rosenpass.

Oh and actually we should use mlock even for secret memory allocated without memfd-secret

from rosenpass.

Oh and actually we should use mlock even for secret memory allocated without memfd-secret

mlock is used- just that it is used by memsec only for the secret, and not for the guard pages. With memfd_secret, all memory is mlocked by default, including the guard pages we add.

from rosenpass.

Oh and actually we should use mlock even for secret memory allocated without memfd-secret

mlock is used- just that it is used by memsec only for the secret, and not for the guard pages. With memfd_secret, all memory is mlocked by default, including the guard pages we add.

We could allocate the guard pages using mmap and use memfd-secret for the secret only?

from rosenpass.

Oh and actually we should use mlock even for secret memory allocated without memfd-secret

mlock is used- just that it is used by memsec only for the secret, and not for the guard pages. With memfd_secret, all memory is mlocked by default, including the guard pages we add.

We could allocate the guard pages using mmap and use memfd-secret for the secret only?

I'm not sure if we have that much control over neighboring memory to memfd-secret since:

memfd_secret() creates an anonymous RAM-based file and returns a file descriptor that refers to it. The file provides a way to create and access memory regions with stronger protection than usual RAM-based files and anonymous memory mappings. Once all open references to the file are closed, it is automatically released.

Please do correct me otherwise

from rosenpass.

Oh and actually we should use mlock even for secret memory allocated without memfd-secret

mlock is used- just that it is used by memsec only for the secret, and not for the guard pages. With memfd_secret, all memory is mlocked by default, including the guard pages we add.

We could allocate the guard pages using mmap and use memfd-secret for the secret only?

I'm not sure if we have that much control over neighboring memory to memfd-secret since:

memfd_secret() creates an anonymous RAM-based file and returns a file descriptor that refers to it. The file provides a way to create and access memory regions with stronger protection than usual RAM-based files and anonymous memory mappings. Once all open references to the file are closed, it is automatically released.

Please do correct me otherwise

Well, you do use a mmap call to actually map the secret into memory…and there you have control over the memory layout…

from rosenpass.

Well, you do use a mmap call to actually map the secret into memory…and there you have control over the memory layout…

True but is there control to allocate memory outside the file descriptor (the memfd that is opened)? I'm not sure I understand how

Unless you are suggesting using munlock on the regions of the memory. I'm not sure if that is ok with memfd_secret, will have to try

from rosenpass.

Well, you do use a mmap call to actually map the secret into memory…and there you have control over the memory layout…

True but is there control to allocate memory outside the file descriptor (the memfd that is opened)? I'm not sure I understand how

Unless you are suggesting using munlock on the regions of the memory. I'm not sure if that is ok with memfd_secret, will have to try

Does the guard page even need to be part of the FD? I though the guard page was a page intentionally left blank.

I think the technique goes something like this:

1. Figure out how many pages you need
2. Figure out some space to put these pages in (how does memsec do this?)
3. Allocate the first page from whatever source using mmap by passing it an explicit pointer address
4. Allocate the memfd secret page(s) passing an address explicitly
5. Allocate the trailing page by passing a pointer to an address

from rosenpass.

Well, you do use a mmap call to actually map the secret into memory…and there you have control over the memory layout…

True but is there control to allocate memory outside the file descriptor (the memfd that is opened)? I'm not sure I understand how
Unless you are suggesting using munlock on the regions of the memory. I'm not sure if that is ok with memfd_secret, will have to try

Does the guard page even need to be part of the FD? I though the guard page was a page intentionally left blank.

I think the technique goes something like this:

1. Figure out how many pages you need
2. Figure out some space to put these pages in (how does memsec do this?)
3. Allocate the first page from whatever source using mmap by passing it an explicit pointer address
4. Allocate the memfd secret page(s) passing an address explicitly
5. Allocate the trailing page by passing a pointer to an address

Thanks I'll try this out today

from rosenpass.

Also, love the idea of moving the McEliece pubkey to the heap--@Prabpreet, would you want to include that in your PR, or would it be easier for me to toss up a separate tiny PR for that and then rebase once you merge yours?

Please feel free to open a separate PR- and for the future- simply call it that you are making the change, and never hesitate to contribute!

from rosenpass.

On it, thanks so much!

from rosenpass.