divergentdave / alice-sled Goto Github PK

It would be nice if ALICE wrote out interesting crashed directories that exhibit vulnerabilities it identifies. Thus far, I have been modifying alicedefaultexplorer.py, by replacing shutil.rmtree(dirname) with if retcode == 0: shutil.rmtree(dirname), killing the program before it can reach shutil.rmtree(alice_config['scratchpad_dir']) in alice-check, and then digging around in /dev/shm.

When running a crash test, the call to unw_init_remote can fail with UNW_EBADREG, presumably because the child process receives SIGKILL and terminates, and alice-strace will exit with a fatal error message. So far, I have worked around this by setting really_output_stacktrace to 0 in syscall.c, but this gets rid of all stack traces. While the method names aren't terribly informative as it is, it would be nice to gracefully handle this error, and just not collect a stack trace for this one process or syscall.

ALICE has plenty of interesting information in its traces beyond what's printed in the list of disk ops. For example, there are unused optional arguments to print_ops to show thread IDs and times. Here's an ad-hoc script I used to examine a trace, this might be worth turning into a command-line tool.

#!/usr/bin/env python
import os
import pprint

from alice import Replayer
from alicedefaultfs import defaultfs

traces_dir = "/home/david/Code/alice-sled/cases/random_ops/traces_dir"
workload_dir = "/home/david/Code/alice-sled/cases/random_ops/workload_dir"
starting_wd = "/home/david/Code/alice-sled/cases/random_ops"
scratchpad_dir = "/dev/shm/alice-inspect"
alice_config = {
    "strace_file_prefix": os.path.join(traces_dir, "strace.out"),
    "initial_snapshot": os.path.join(traces_dir, "initial_snapshot"),
    "base_path": workload_dir,
    "starting_cwd": starting_wd,
    "ignore_mmap": False,
    "scratchpad_dir": scratchpad_dir,
}
replayer = Replayer(alice_config)
replayer.set_fs(defaultfs('count', 1))
pprint.pprint(replayer.micro_ops[75].__dict__)

After fixing the libunwind issue, it appears that there are still multiple ways alice-strace will fail if a tracked executable receives SIGKILL. I noticed one that appeared to be happen while dumping data from a buffer, and one error message from somewhere else. Each of these appears to be individually rarer than the unw_remote_init issue, based on the output.

Resources:

• "Finding consistency violations in databases is an NP-complete problem – the cost grows exponentially with the number of transactions." --https://www.scylladb.com/2020/12/23/jepsen-and-scylla-putting-consistency-to-the-test/
• https://mastodon.jepsen.io/@jepsen/105470006648993036
• https://aphyr.com/posts/331-jepsen-voltdb-6-3
• https://aphyr.com/posts/314-computational-techniques-in-knossos
• https://jepsen.io/consistency
• https://blog.acolyer.org/2020/11/23/elle/
• https://www.usenix.org/sites/default/files/conference/protected-files/osdi20_slides_tan.pdf
• https://www.usenix.org/system/files/osdi20-tan.pdf

It would be useful to have a test case where an extra binary prepares a dirty database, before the ALICE workload operates on it, to improve coverage. This would allow exploring databases where recovery has been run multiple times, and would allow working on larger databases without paying the cost of writing the entire database from within ALICE's polynomial blowups.

ALICE was written in Python 2, which is on borrowed time. Converting to Python 3 might be worthwhile. BitVector supports Python 3, which helps. Since ALICE deals with binary data as str currently, the transition will require some careful rewriting to use bytes and str as appropriate.