Segmentation fault when application stops (Ruby 3.2.2, about mini_racer HOT 11 OPEN

Here’s a small Dockerfile which reproduces this issue on both amd64 and arm64:

FROM ruby:3.2.2-slim-bookworm

RUN \
  apt-get update --yes && \
  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    build-essential libjemalloc2 && \
  ln -s /usr/lib/*-linux-gnu/libjemalloc.so.2 /usr/lib/libjemalloc.so.2
ENV LD_PRELOAD=/usr/lib/libjemalloc.so.2

RUN gem install mini_racer

CMD ruby -r mini_racer -e 'p MiniRacer::Context.new.eval("1+2")'

I did some testing with other base images and it seems like ruby:3.1-slim-bullseye with libjemalloc2 5.2.1-3 works fine but ruby:3.1-slim-bookworm with 5.3.0-1 does not. Installing the older jemalloc into Bookworm did not work so presumably it must be something else other than the jemalloc version?

from mini_racer.

Hey @ukolovda 👋

Can you provide us with some more information? Most importantly: Can you reproduce it? And if so, can you provide a minimal example?

from mini_racer.

ah, jemalloc 😞

We had several Ruby and mini_racer issues with jemalloc in the past, e.g. #242.

I think that might be something @lloeki or @SamSaffron can take a look?

from mini_racer.

I use jemalloc too...

@jasoncodes , thank you!

from mini_racer.

I'm wondering whether it would get confused at times as to which malloc/free symbol is which depending on how the ruby jemalloc linking happens (I think glibc is always dynamic, dunno about musl).

https://github.com/jemalloc/jemalloc/wiki/Getting-Started

from mini_racer.

Oh wait, I didn't see that:

ENV LD_PRELOAD=/usr/lib/libjemalloc.so.2

So you're injecting jemalloc, not linking Ruby against jemalloc at Ruby build time? Do you get the same result with a Ruby build against jemalloc?

from mini_racer.

Do you get the same result with a Ruby build against jemalloc?

Good question. Looks to work fine if the Ruby is built using jemalloc instead of injecting.

I tested this using the first Ruby 3.2.2 w/ jemalloc base image I found which I could confirm has a working jemalloc with MALLOC_CONF=stats_print:true ruby -e exit:

FROM jiting/ruby:3.2.2-slim-bookworm
RUN apt-get update && apt-get -y install build-essential
RUN gem install mini_racer
CMD ruby -r mini_racer -e 'p MiniRacer::Context.new.eval("1+2")'

from mini_racer.

Okay so I think what might happen is:

• libv8 is built against glibc and links dynamically to it
• This usually resolves the dynamic malloc+free symbols to glibc
• When LD_PRELOAD is used to hook jemalloc in it applies to Ruby but it also applies to any other thing that dynamically links to libc's malloc+free (which would be any Ruby extension having linked to glibc)
• We have mini_racer_loader that binds symbols with RTLD_LOCAL and RTLD_DEEPBIND when dlopening mini_racer_extension, this is supposed to make mini_racer_extension symbols hidden from outside and prefer its own internal symbols, but there is no malloc+free inside, so they resolve to glibc's malloc+free.
• Even if there were malloc+free internal to mini_racer_extension , depending of how linking is done inside the mini_racer_extension shared library for these symbols (e.g malloc+free are internal but dynamic symbols) it might be so that LD_PRELOAD takes precedence over RTLD_DEEPBIND. Not the case here but we can entertain the thought.
• Thus with LD_PRELOAD, libv8 will use jemalloc instead of glibc's malloc+free
• libv8 does not like that, hilarity ensues

When ruby is built statically against jemalloc, the malloc+free symbols are entirely static (and might even be elided entirely, instead replaced by pure addresses), in which case they are unavailable to extensions. Thus Ruby can do its things with jemalloc and libv8 its things with glibc, and since ownership is split (nothing allocated from one is ever freed by the other) they both live happily ever after.

The remaining question is why libv8 would not like having its allocator swapped with jemalloc via LD_PRELOAD but I guess that's an upstream libv8 question.

from mini_racer.

This morning I came across docker-library/ruby#182 (comment) which suggests using patchelf instead of LD_PRELOAD:

FROM ruby:3.2.2-slim-bookworm

RUN \
  apt-get update --yes && \
  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    build-essential libjemalloc2 patchelf && \
  patchelf --add-needed libjemalloc.so.2 /usr/local/bin/ruby

RUN gem install mini_racer

CMD ruby -r mini_racer -e 'p MiniRacer::Context.new.eval("1+2")'

This seems like a nicer way to patch an existing Ruby as it avoids affecting all processes. Unfortunately it doesn’t solve the problem here as dynamic linking still results in libv8 using jemalloc and crashing on exit. I’m guessing within the single Ruby process, patchelf and LD_PRELOAD are effectively the same thing.

Only option right now to use mini_racer with jemalloc seems to be building Ruby with jemalloc.

from mini_racer.

Only option right now to use mini_racer with jemalloc seems to be building Ruby with jemalloc.

At least it seems to be more reliable. I'm running with LD_PRELOAD in production for a while and haven't had an issue so far; I'm still using ruby:3.2.2-bullseye (docker) though.

Since this is popping up again and again over time, I'm wondering if we should add something to the troubleshooting section of the README to recommend not using jmalloc for now, or use Ruby statically build against it. What do you think, @lloeki?

from mini_racer.

FYI Rails has merged a PR to LD_PRELOAD jemalloc in their default Dockerfile template: rails/rails#50943.

In lieu of something in the README, may I suggest renaming this Issue to mention jemalloc?

from mini_racer.