central crashes at startup with quay.io/stackrox-io/main:3.72.1 (also: 3.72.0) about stackrox HOT 15 CLOSED

Hello folks,

thank you for reporting this incident, and our sincere apologies for the disruption this caused.

Due to an unexpected schema change in an upstream vulnerability feed, a corrupted CVE data file has been published to https://definitions.stackrox.io/ and consumed by a large number of Central instances. As a result of the data corruption, Central crashes. To the best of our knowledge, this affects all Central versions.

While we have already taken steps to ensure a non-corrupted version is served from https://definitions.stackrox.io/, affected Centrals will not be able to get out of this crashloop state without manual intervention in order to delete the previously downloaded, corrupted file.

In order to get Central back to a working state, please follow these instructions to delete the file: https://gist.github.com/misberner/c43a666fc0a6ff335925b9800473d489

We have already identified further steps to prevent issues of this kind from happening in the future, specifically ensuring that Central self-heals when a dependency corruption is fixed.

from stackrox.

Same issue happening with version 3.70.1 without any recent changes in configuration/version.

from stackrox.

Thank you for reporting! The team got notified about this issue by internal monitoring and is currently looking into this.

from stackrox.

Working like a charm! Thank you very much for the quick workaround.

from stackrox.

Thanks for providing information about the issue.
The team has found the root cause and currently working on releasing a workaround and releasing a patch fix as soon as possible.

from stackrox.

Unfortunately, all versions of Central may be affected. We understand the root cause and have a recovery command on the way. It will be shared here soon. Sorry for the downtime.

from stackrox.

Same here on 3.71.0.

Looking at the stack trace I'm guessing something has changed in an external vulnerability database which is now making the Stackrox processing crash? And that's why it started happening suddenly without any changes and affects several versions.

from stackrox.

Works for me as well. Thank you for the fix. 🙂

from stackrox.

confirm it works

from stackrox.

Same issue. Running Version 3.72.0. Quick fix would be appreciated.

from stackrox.

thanks for reporting @iljaweis
we have the same

from stackrox.

Same problem.
image: 'registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8:3.71.0'
It worked for more then month and then suddenly falled into crash loop with the same error.

from stackrox.

thanks for the response
we've encountered the same problem on 3.71.0

from stackrox.

For posterity: the fix for the problem has been merged, #3504

from stackrox.

Thanks so much for the fix ;)

from stackrox.