memcache: server error about nds HOT 12 CLOSED

This doesn't sound like a nds specific error. NDS usually emits errors nds: [error type]. Is it consistent or intermittent? In any case, nds is designed to work fine with any type of memcache error or even if the memcache backend completely goes down.

The one caveat is that nds.Delete needs to ensure that memcache.Delete completes successfully before it works so that it can guarantee strong cache consistency.

Please let me know if you find out any more information. I'll leave this issue open a bit longer. My instances haven't logged this error today but I'll keep looking.

from nds.

Sorry, the dev who passed on the log didn't give me the whole thing. This is what was getting thrown.
nds:lockMemcache AddMulti memcache: server error

It was happening on every datastore request, and I haven't seen anything specific that is different that might have caused it. Is there maybe a difference in max record sizes between memcache and datastore?

from nds.

Looks like this may be tied to having special characters in the key. I tried 3 specifically: ?, =, %. All 3 cause the memcache: server error to be thrown. I took those out, it worked just fine.

from nds.

@josephbergevin that's interesting to know, however I don't think this is an issue affecting NDS. The memcache keys will never use those characters. datastore.Key.Encode() uses base 64 encoding and removes the trailing "=". This is identical to what Python's ndb does.

@derekperkins are you still seeing these problems consistently? Could it have been that the API was down at the time?

from nds.

@jongillham I actually work in the same room as @josephbergevin, so yes, I'm seeing the same error. :) I think Joe narrowed the problem down to the length of the StringID.

from nds.

I investigated further and it seems the error may be tied to a character limit on the StringID. 147 chars is the max limit I could use in a key. If I go over that, I get these errors/warnings:
GetMulti gives a warning: nds:lockMemcache AddMulti memcache: server error
PutMulti throws an error: memcache: server error

from nds.

@josephbergevin awesome deductive work! I see the Python and Go SDK functionality differs here. The max key size on both platforms is 250 bytes. However Python automatically hashes the key if it is longer than 250 bytes and Go just raises an error.

This error must be what you are seeing. I'll investigate what type of hashing Python does and then copy it. I suppose there will be risk of hash collision but it must be small enough for the guys that made the Python SDK to not worry about.

What do you think?

from nds.

Just for reference here's what Python's API does.

MAX_KEY_SIZE = 250

if len(server_key) > MAX_KEY_SIZE:
    server_key = hashlib.sha1(server_key).hexdigest()

I wonder if it is worth using sha256 or sha512 to reduce the possibility of collisions...

http://stackoverflow.com/questions/1867191/probability-of-sha1-collisions

Probably not.

from nds.

Yeah, probably not. For it to be an issue, all of these would have to be the case:

1. The 2 StringIDs would have to hash to the same key
2. Both of the total keys have to exceed 250 characters
3. Both would have to be called in close enough proximity that neither one has been evicted from memcache already.

If it were more persistent data, it might be more of an issue, but I don't forsee this being that big of an problem.

from nds.

Yes, I agree with @derekperkins. That sounds great.

from nds.

Commit 924699e should hopefully fix this issue. I'll await feedback before I close this as the dev environment does not replicate this issue.

from nds.

That worked! I think you can go ahead and close the issue. Thanks for the quick responses!

from nds.