Performance regression from 6.4.0 -> 7 about bazel HOT 8 OPEN

--experimental_remote_merkle_tree_cache_size=1000 is potentially too small (see #18686). Can you bump that up to something like 10000 and see if the issue goes away? (Unfortunately it's difficult to determine a good value; that's one of several problems with the Merkle tree cache as explained at length in #21378.)

from bazel.

Understood, I will try that. From that discussion, should I potentially disable the merkle cache altogether?

Also related, I am still observing memory leaks and OOMs as tracked in #16913 on 6.4.0. I have been unable to have successful stable bulds on 7.x as to determine if the memory leak is resolved or not due to your changes the merkle caching

from bazel.

Understood, I will try that. From that discussion, should I potentially disable the merkle cache altogether?

In my experience, the cache is only helpful if your build is dominated by large tree artifacts (i.e., declare_directory) and runfiles trees; for everything else, it's a pessimization. So I'd definitely do some measurements to determine whether it helps or hinders.

from bazel.

We are very mixed; some areas in code have small files & trees (golang, python, java) and others have large directories from node_modules, managed by rules_js

I'll try with and without to see if there is a benefit. How much does experimental_remote_merkle_tree_cache_size affect the heap usage, is there a static memory usage per cache size? I'm cautious to bump it up too high and result in OOMs.

from bazel.

Unfortunately, there's no straightforward connection between the flag value and the amount of heap consumed; each cache entry corresponds to a depset node, and has size proportional to the number of direct elements of that node (roughly num_files * (length(basename) + length(digest) + overheads)). If one of the direct elements is a tree artifact or a runfiles tree, it's promoted to its own cache entry with the corresponding number of files.

from bazel.

I can confirm that --noexperimental_remote_merkle_tree_cache allows the build to progress normally. Did the implementation of that merkle tree caching change in 7.x? I do not see the same performance issue in 6.4.0, and wondering if some regression was introduced.

from bazel.

Yes, I believe an unintentional regression was introduced in the lead-up to 7.x, causing a catastrophic slowdown when the cache size is too small to fit the largest depset in the build (in 6.x, a cache too small might have been slower, but not catastrophically so). As long as the size is large enough, I haven't seen evidence that 7.x is slower than 6.x.

Unfortunately, it's difficult to revert the culprit (it's actually a combination of two different changes) because some later work has come to depend on it, and per the discussion in #21378, we'd rather spend time rearchitecting the Merkle tree cache instead of addressing performance issues with the current implementation.

from bazel.

This makes complete sense, and now I understand why 6 was sufficient, even if less optimal than it could be. Thank you for explaining.

I will run some experiments on my end on 6.x and 7.x to evaluate the effectiveness of --experimental_remote_merkle_tree_cache if the cache size is adequate. Hopefully I won't want to object to your proposal of removing the cache altogether 😄

from bazel.