cache_key already exists errors in PostgreSQL about django-ratelimit HOT 8 CLOSED

I have the same issue using a Postgres cache.

from django-ratelimit.

I'm curious if the ratelimiter still works or if these errors are more of an annoyance/noise for logs?

When we de-pickle the values, they're single integers, like 1, 2, 3, etc.

Those are the usage counts.

I don't know that this is fixable. The first thing ratelimit tries to do is add the new key to the cache, and whether that succeeds or fails is important for the rest of processing the request. It also looks like the database backend doesn't have an atomic increment—and since it's storing encoded, pickled values, I'm not sure that it could.

Given the lack of atomic increment, I'm not sure it's reasonable to call the database backend "supported" for use with this library. Races will potentially allow multiple transactions to commit the same count number.

I do think it might be possible to build a "partial" implementation of the cache interface base—one that only supports integer values and so can support atomic/transactional increment—based on a dedicated model. That would require its own expired-row-reaping cron, too, since it wouldn't be in the usual cache table. I'd be very open to including an implementation like that, maybe in django_ratelimit/contrib/modelcache or something similar, but I'm probably not going to take it on myself.

(Theoretically there are a few ways to make this library work but it requires a storage mechanism with either atomic increment or check-and-set to work reliably.)

At the very least, I will add something to the documentation, though, about requiring a cache backend with support for an atomic increments to work reliably.

from django-ratelimit.

Realistically should not be using database as cache for something like this

from django-ratelimit.

have the same issue

from django-ratelimit.

That would require its own expired-row-reaping cron, too, since it wouldn't be in the usual cache table

I'm not sure it would be required... couldn't you just could have a second ORM call that reaps expired rows as part of the same thread of execution that does the atomic insert on the model table?

from django-ratelimit.

Realistically should not be using database as cache for something like this

I get why you're being downvoted, but I agree. If you're using a rate limit to protect your DB from crashing, then storing the rate limit in the DB makes no sense, because no DB == no rate limit.

from django-ratelimit.

no DB == no rate limit.

no DB => no website

You might be using ratelimiting to protect against other types of annoying behavior, such as sign-up spam or a lot of form submissions.

from django-ratelimit.

That would require its own expired-row-reaping cron, too, since it wouldn't be in the usual cache table

I'm not sure it would be required... couldn't you just could have a second ORM call that reaps expired rows as part of the same thread of execution that does the atomic insert on the model table?

True, maybe not required but definitely preferable to doing several queries to check the ratelimits before we even get to the view.

I added notes to the docs in #265 highlighting the need for atomic increments.

I'm still open to either contrib-style implementation of the partial cache interface, or other ideas or recipes to add to the docs, if anyone is interested in pursuing those. For now, I'm going to close this out and say that the DB cache backends and any others without atomic increment operations aren't supported.

from django-ratelimit.