Crashing with "panic: failed to migrate ds" about rudder-server HOT 8 OPEN

Hi @shashank42 , thank you for reporting the issue. I will get back to you soon.

from rudder-server.

@shashank42 can you please share the version number of rudder-server you're using?

from rudder-server.

For the current deployment

    Image:         rudderlabs/rudder-server:latest
    Image ID:      docker.io/rudderlabs/rudder-server@sha256:6ed8c0155ed821dfded3ba8efd6909b313989684cacfe210322f4131ecdbbe77

https://github.com/glip-gg/rudderstack-helm

Here is the repo I am using

from rudder-server.

I think the issue is similar to this. I have taken the steps here, that is deleting all the tables inside jobsdb, however the errors have not stopped
https://rudderstack.slack.com/archives/C01E4PLB135/p1692320392539509?thread_ts=1689977868.588139&cid=C01E4PLB135

from rudder-server.

@shashank42 does the issue persist when you restart the postgres db pod?
The shared logs are not enough to make any conclusion - it seems migration loop timeouts, and from the db logs it looks like the database is overloaded

from rudder-server.

I just did a new deployment on a new cluster to check fresh setup case. There are still restarts, but less frequent.

Postgress doesn't seem to run out memory at any point.

from rudder-server.

@shashank42 can you check the resource (CPU, RAM) usage before and after you start the server? please also mention what resources(CPU, memory) have you provisioned for postgres?

Context: with the limited info, we are guessing that enough resources might not have been allocated (assuming the server is not overloaded with unusual number of requests) which causes a query timeout

from rudder-server.

@shashank42 can you please help with more information as mentioned above?

from rudder-server.