Fail starting up if ulimit is too low about readyset HOT 8 CLOSED

Moved to a different Project.

from readyset.

@Gautam Gopinadhan

from readyset.

Awesome! All that sounds great. 😀

from readyset.

@imeyer yes, that was the approach I was going to take at startup

Makes sense about providing precise guidance being tricky. Right now the signature we see is "I/O Error: Too many open files", which happens hours into readyset being up. So even a slightly more error sooner (e.g. "System misconfiguration detected: ulimit -n is set too low $current_value. ReadySet may need up to $recommended_value file descriptors.") and letting operators figure out how to fix the error themselves would probably prevent most of the pain in this case.

from readyset.

Was your plan to run a ulimit -Hn && ulimit -Sn from within readyset itself? I think the most/closest would be to get the runinng PID of readyset immediately at startup (in the readyset Rust code), and parse the /proc/<pid>/limits file for that information, and that would cover pretty much all cases I mentioned. (i'm sure there is an idiomatic rust way, I just don't know it 😄)

The tricky part is informing the user how to set it appropriately based on how they're running readyset as limits across distros etc are not uniform. For example the defaults for the following distros…

Amazon Linux 2: sysctl fs.file-max && ulimit -Hn && ulimit -Sn
fs.file-max = 9223372036854775807
32768
32768

Ubuntu 22.04: sysctl fs.file-max && ulimit -Hn && ulimit -Sn
fs.file-max = 9223372036854775807
1048576
1024

Fedora 38: sysctl fs.file-max && ulimit -Hn && ulimit -Sn
fs.file-max = 9223372036854775807
524288
1024

Are quite different!

from readyset.

@imeyer From my reading, for 1,2,and 4 we should still be able to detect if readyset won't have enough file descriptors by running ulimit -n on startup, but I'm not sure about (3). Do you have a recommendation for how to detect the file descriptor configuration in that use case?

from readyset.

Some things to consider for limits precedence (at least on Linux, unclear on the BSDs)

1. sysctl is the facility for kernel tunables (fs.file-max takes precedence over values set via limits.conf(5))
2. PAM controls user/group level limits and can not exceed those defined by the kernel
   1. There are hard and soft limits. ulimit -Hn and ulimit -Sn respectively
   2. Limitations set via PAM are per process (i.e. you run readyset from your logged-in terminal), they are not global to a user/group/session.
3. systemd has it's own mechanism for setting limits via LimitNOFILE=<int> in the [Service] section of the unit file.
4. can be set before the process is started without changing anything in limits.conf(5) provided it does not exceed the hard limit.. (e.g. ulimit -n 32768 && readyset … )

from readyset.

Lets avoid getting fancy and just request for ulimit to be set to a large number (32K), if and its not set, we will fail to start up.

cc: @emrysal - we should set the ulimit in the docker scripts if this is not already happening.

cc: @coredb-service-userstanton - Similar setting may be needed in the helm installer as well.

from readyset.