Sentry requesting unknown symbols about sentry HOT 9 OPEN

Assigning to @getsentry/support for routing ⏲️

from sentry.

Routing to @getsentry/product-owners-issues for triage ⏲️

from sentry.

I am going to have the processing team take a look at this on Monday

from sentry.

Hi, thank you for the thorough report. What immediately strikes me as odd is that Symbolicator requesting files that aren't found on the server (whether because they weren't uploaded or because they're entirely fictitious) shouldn't trigger host blocking at all—it's only supposed to happen when a download fails or times out. Do you perchance have error logs from before the log message you posted about the host being blocked?

If you don't need the block list feature at all, a bandaid fix would be to set deny_list_threshold to an unreasonably high number in your config so no host ever gets blocked. But we should still 1) make a proper option to switch it off and 2) find out if there is a logic bug that makes it block hosts it shouldn't.

from sentry.

I opened getsentry/symbolicator#1407 in the Symbolicator repo that might address your problem in the immediate term:

1. It lets you switch off the denylist entirely by setting deny_list_enabled to false.
2. If you still want to use the denylist and just don't want it to block your own symbol server, you can add the URL to deny_list_never_block_hosts.

from sentry.

Maybe this is already possible in the logs, and I just haven't pieced it together yet.

I believe we are already capturing those errors. In general such requests would have to fail repeatedly for it to be blocked.

from sentry.

Sorry, I didn't get a notification y'all had responded. Attached is a log from docker with more context up to the block:

Looking through it now, it actually looks like symbolicator started receiving certificate errors. We had seen these before, and I switched everything back from https to http to avoid them while we work it out on our end (we have self-signed certs, and despite adding them to the docker container), but it looks like at some point last week it went back to the old https endpoint. However, I can't see where that's actually being set.

In our project configs, it seems to have the correct settings

But then looking at the logs, it's requesting this:

Fetching debug file from https://tools.REDACTED.com/symbolcache/symbols/REDACTED/90/c2362b9d1f1f5088abfa3bde69baaf1/debuginfo

log.txt

from sentry.

Somewhat related, is there a way to request Sentry try to re-symbolicate a given crash? I.E., if I get these issues fixed, and assuming all of the symbols are still around, it would be nice to go back to some of these reports and ask Sentry to try again.

from sentry.

That would work via reprocessing, although I'm not sure how that works on self-hosted.

from sentry.