appview-team / appview Goto Github PK

On aarch64 machines, go processes crash intermittently with a "bad g". This has only been seen on aarch64. This was uncovered in go integration tests (go_20) in test cases for "signalHandlerStatic" and "signalHandlerStaticStripped". By adding a go routine to constantly write to the console, these test now fail within a small number of iterations (on this branch). To see them ASAP, I'd recommend commenting out the other test cases and running the go_20 signalHandler tests in a loop to see the "bad g" failure.

We may not have a perfect understanding, but we believe that when a signal happens while we are on the switched stack (for pcre2), go tries to retrieve the "g" from the stack. It's not going to find "g" on the stack when we're currently on our own stack executing pcre2 code, so go crashes. Without the new go routine that writes to the console, the "bad g" happens very infrequently, but importantly "bad g" in the signalHandlerStatic tests has also been observed on branches that do not have the stack pool implementation here. To the best of our knowledge, we could see this any time a signal handler interrupts us while we're running our c code.

If this description is correct, we think a possible solution is to mimic how go knows if c code is currently running.
https://github.com/golang/go/blob/master/src/runtime/cgocall.go
https://github.com/golang/go/blob/master/src/runtime/asm_arm64.s

I have used ld.so preload with rules file for nginx and started the nginx container with podman. While the container is able to start it is not scoped.

It is now possible for libscope.so to create a core dump in release 1.3 on glibc systems.

The current implementation (coredumper) has some discovered limitations:

• musl is not supported by coredumper
• go is not supported by coredumper
• applications running on ubuntu 22 are not supported by coredumper

On the topic of coredumps, there are a number of other things that may need some consideration in the future:

1. coredumps themselves need some context to be useful. By context, I mean the application itself and dynamic libraries it references. Do we need to have an option to capture all of this information too (for apps running in a container?)
2. we currently don't attempt to transfer coredumps across a network connection, do we need to add this?
3. we've seen coredumps in our java7 integration (debian 8) test take 60s to complete. Otherwise the creation seems to be very responsive. Does this merit further investigation? I'm inclined to think we should wait to see if we observe this anywhere else.
4. Right now in snapshotSignalHandler(), we're delaying 2s before calling the appSignalHandler. And the daemon waits 1s after receiving a signal before trying to grab the output files created by the library. The synchronization between libscope.so and scope daemon should probably be improved. @michalbiesek has shown that if a process joins the same mount namespace as the crashing process, it can access to the any file in that namespace even after all other processes in that namespace have exited. (though there are limitations about searching or viewing directories after the container has stopped). Maybe this could help?

$ /bin/linux/aarch64/scope run -- lxc ls
ERROR: ld.so: object 'libscope.so' from /etc/ld.so.preload cannot be preloaded (failed to map segment from shared object): ignored.
(same result with LD_PRELOAD or ld.so.preload)

$ LD_DEBUG=libs lxc ls
results in numerous symbol errors. a few examples:
calling init: /lib/libscope.so

 20015:	/lib/libscope.so: error: symbol lookup error: undefined symbol: SSL_read (fatal)
 20015:	/lib/libscope.so: error: symbol lookup error: undefined symbol: SSL_write (fatal)
 20015:	/lib/libscope.so: error: symbol lookup error: undefined symbol: SSL_get_fd (fatal)

 20015:	/lib/libscope.so: error: symbol lookup error: undefined symbol: PR_SetError (fatal)

 20015:	/lib/libscope.so: error: symbol lookup error: undefined symbol: uv__read (fatal)

 20015:	/lib/libscope.so: error: symbol lookup error: undefined symbol: uv_fileno (fatal)

other libs exhibit symbol errors, such as librt. however, are not preloaded.

Steps To Reproduce

root@precision:/home/sean/sandbox# scope attach dockerd
WARNING: Session history will be stored in /root/.scope/history and owned by root
Attaching to process 3029
root@precision:/home/sean/sandbox# scope metrics
NAME             VALUE              TYPE     UNIT           PID     TAGS
proc.start       1                  Count    process        3029    args: /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.s…
proc.cpu         3.990642489e+09    Count    microsecond    3029    host: precision,proc: dockerd
proc.cpu_perc    39906.42489        Gauge    percent        3029    host: precision,proc: dockerd
proc.mem         3.905616e+06       Gauge    kibibyte       3029    host: precision,proc: dockerd
proc.thread      42                 Gauge    thread         3029    host: precision,proc: dockerd
proc.fd          25                 Gauge    file           3029    host: precision,proc: dockerd
proc.child       41                 Gauge    process        3029    host: precision,proc: dockerd
root@precision:/home/sean/sandbox# scope events
Empty event file.

Environment

- AppScope: 1.3.1
- OS: ubuntu 22.04
- Architecture: x86
- Kernel: 5.19.0-38-generic

Early versions of Go use Go strings. Later versions of Go do not use Go strings. Therefore, the free operation is not needed in later versions. However, we should still put the free function back for older versions.

Steps To Reproduce

Scoping following container fails
Dockerfile.redis:

FROM redis:6

COPY --from=cribl/scope:1.3.2 /usr/local/bin/scope /usr/local/bin/scope
RUN /usr/local/bin/scope extract /usr/local/lib/

ENV LD_PRELOAD="/usr/local/lib/libscope.so"

# Expose Redis port
EXPOSE 6379

# Start Redis server
CMD ["redis-server"]

docker build --file Dockerfile.redis --tag myredis:latest .
docker run myredis:latest

There is no libscope.so in the redis process above

Scoping same container using different image works fine:

Dockerfile.alpine:

FROM redis:6-alpine

COPY --from=cribl/scope:1.3.2 /usr/local/bin/scope /usr/local/bin/scope
RUN /usr/local/bin/scope extract /usr/local/lib/

ENV LD_PRELOAD="/usr/local/lib/libscope.so"

# Expose Redis port
EXPOSE 6379

# Start Redis server
CMD ["redis-server"]

docker build --file Dockerfile.alpine --tag myredisalpine:latest .
docker run myredisalpine

Environment

- AppScope: 1.3.1
- OS: Ubuntu
- Architecture: x86_64
- Kernel: 5.19

When we read a file with this test app (calling read or __read_chk), I observed that we do not produce an fs.read metric event. Also, in the fs.close event associated with the file read, file_read_bytes does not change from 0.

#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>

#define SIZE 512
static int fd;
static char buf[SIZE];

void verify_read(void) {
	//__read_chk(fd, buf, SIZE, SIZE);
	read(fd, buf, SIZE);
}

void setup(void) {
	memset(buf, '*', SIZE);
	fd = open("testfile", O_RDWR | O_CREAT, 0700);
	write(fd, buf, SIZE);
}

void cleanup(void) {
	if (fd > 0) close(fd);
}

int main() {
	setup();
	verify_read();
	cleanup();
}

We do see expected results using other commands:

SCOPE_EVENT_METRIC=true ./bin/linux/x86_64/scope head [some file]
scope events

npm 8.5.1
ubuntu 22.04

$ scope npm install express
$ /usr/bin/npm is not a viable ELF file

The child node process is successfully scoped, but because npm runs a script, AppScope throws a warning about the file format.