Rewards are not payed out consistenly about stacker.news HOT 5 CLOSED

No need to randomize them. Just offset them so nothing that might take a long time runs at the same time.

We can't retry rewards as-is. They aren't atomic.

from stacker.news.

No need to randomize them. Just offset them so nothing that might take a long time runs at the same time.

Also good idea. I forgot that we don't have the exact same situation here. At my previous position, I had the idea to randomize since we never knew when our servers are going to get hit by a huge wave of automated abuse reports in bulk, not realizing they are actually DDOSing our servers, trying to process all these bulk abuse reports at once, lol. here, we don't have such consistent load with huge spikes of incoming requests, I guess

We can't retry rewards as-is. They aren't atomic.

Oh, right, that would be a bad idea, lol

from stacker.news.

Yeah this appears to be caused by us timing out waiting for a connection from prisma's connection pool. I'd guess the worker picked to run rewards is occasionally running other long running tasks. Both rewards and historical materialized views are both scheduled to run at exactly midnight.

from stacker.news.

So solution might be to just add retry (for now)? I think you mentioned that to me once

I used to have similar problems at my previous position and it took us really long to fix.

One solution I brought up was to randomize when the queries run so not all our REFRESH MATERIALIZED VIEW CONCURRENTLY want to start at exactly the same time but we never did that for some reason

So maybe just do this? Randomize the query starts? Like +-1 hour?

from stacker.news.

This specific issue should be fixed in 628f755

from stacker.news.