Exception on RemoveLeastRecentlyUsedCacheItemWhenOverMaxCacheSize, null key about service-fabric-distributed-cache HOT 8 CLOSED

I have a Pull Request (#14), but I'm worried that this duct-tapes something fundamental on initial RunAsync(). Please let me know your thoughts.

from service-fabric-distributed-cache.

Thanks @Ryanman. We will look at your pull request and let you know what we think.

from service-fabric-distributed-cache.

I see the PR was approved. I'd like us to keep this issue open until I'm able to report back that a) this solves the issue and b) that the cache is able to recover from being replicated. With the time this is occurring it may be a day or two.

If we're doing something wrong to cause this hopefully I'll have info for that as well.

from service-fabric-distributed-cache.

Sounds good. I was able to get the error in my local testing when I set the MaxCacheSizeInMegabytes be a negative value so that I could get all cached items to all be removed. With your fix this does not happen even if the MaxCacheSizeInMegabytes is set to a negative value.

from service-fabric-distributed-cache.

Hey all - Ran some tests last night. Our two different environments encountered the issue around 6 hours after our deployment and test. This is also the length of our longest sliding expiration.

Are y'all able to push the latest update to NuGet relatively soon? There's a chance we'll uncover another issue with some of the data we're putting in with the long sliding expiration, but to be frank we've been loading and retrieving those items for a while now before the issue began. Perhaps there's other changes we've made that have indirectly caused the deletion of expired items to bomb out.

from service-fabric-distributed-cache.

Shifted our longest expiration time to 5 hours instead of six, Failures occurred after 5 hours instead of 6. So this is relatively confirmed to occur when the cache is emptied during long periods of inactivity.

from service-fabric-distributed-cache.

Sorry I thought your PR already went through and Nuget package was pushed but it looks like your test is failing the build:

System.NullReferenceException : Object reference not set to an instance of an object.


Stack trace
   at SoCreate.Extensions.Caching.ServiceFabric.LinkedDictionaryHelper.AddLast(CacheStoreMetadata cacheStoreMetadata, String cacheItemKey, CachedItem cachedItem, Byte[] newValue) in /home/vsts/work/1/s/src/SoCreate.Extensions.Caching.ServiceFabric/LinkedDictionaryHelper.cs:line 77
   at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.<>c__DisplayClass20_0.<<SetCachedItemAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/src/SoCreate.Extensions.Caching.ServiceFabric/DistributedCacheStoreService.cs:line 136
--- End of stack trace from previous location where exception was thrown ---
   at SoCreate.Extensions.Caching.ServiceFabric.RetryHelper.<>c__DisplayClass4_0.<<ExecuteWithRetry>b__0>d.MoveNext() in /home/vsts/work/1/s/src/SoCreate.Extensions.Caching.ServiceFabric/RetryHelper.cs:line 72
--- End of stack trace from previous location where exception was thrown ---
   at SoCreate.Extensions.Caching.ServiceFabric.RetryHelper.ExecuteWithRetry[TResult](Func`3 operation, Object state, CancellationToken cancellationToken, Int32 maxAttempts, Nullable`1 initialDelay)
   at SoCreate.Extensions.Caching.ServiceFabric.RetryHelper.ExecuteWithRetry(IReliableStateManager stateManager, Func`4 operation, Object state, CancellationToken cancellationToken, Int32 maxAttempts, Nullable`1 initialDelay) in /home/vsts/work/1/s/src/SoCreate.Extensions.Caching.ServiceFabric/RetryHelper.cs:line 83
   at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.SetCachedItemAsync(String key, Byte[] value, Nullable`1 slidingExpiration, Nullable`1 absoluteExpiration) in /home/vsts/work/1/s/src/SoCreate.Extensions.Caching.ServiceFabric/DistributedCacheStoreService.cs:line 105
   at SoCreate.Extensions.Caching.Tests.DistributedCacheStoreServiceTest.RemoveCachedItemAsync_RemoveItemsFromLinkedDictionary_RemovalWorksWithMalformedMetadata(Mock`1 stateManager, Mock`1 cacheItemDict, Mock`1 metadataDict, ServiceFabricDistributedCacheStoreService cacheStore) in /home/vsts/work/1/s/tests/SoCreate.Extensions.Caching.Tests/DistributedCacheStoreServiceTest.cs:line 343
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__139_0(Object state)

from service-fabric-distributed-cache.

That's embarassing - my apologies. I got all the test to pass, then "Cleaned up" before opening the PR, with the cachedItems object not having direct follow-up references in the test.

Re-added the necessary memory store setup call: #16

In other news we tested the new code by temporarily adding the package as a project, and the problem appears to be solved in our lower environments.

from service-fabric-distributed-cache.