socreate / service-fabric-distributed-cache Goto Github PK

I'm testing the SetAsync. The code is:

            var options = new DistributedCacheEntryOptions
            {
                SlidingExpiration = TimeSpan.FromSeconds(300)
            };
            await _distributedCache.SetAsync("AccessToken", Encoding.UTF8.GetBytes("55"), options);

It's throwing with the following stack trace.

System.InvalidCastException: Unable to cast object of type 'System.Fabric.SingletonPartitionInformation' to type 'System.Fabric.Int64RangePartitionInformation'.
at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreLocator.<>c__DisplayClass12_0.b__0(Partition p)
at System.Linq.Enumerable.Single[TSource](IEnumerable1 source, Func2 predicate)
at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreLocator.GetPartitionInformationForCacheKeyAsync(String cacheKey)
at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreLocator.GetCacheStoreProxy(String cacheKey)
at SoCreate.Extensions.Caching.ServiceFabric.ServiceFabricDistributedCache.SetAsync(String key, Byte[] value, DistributedCacheEntryOptions options, CancellationToken token)

Any reason there is no option to get all stored keys or remove all stored keys within a cache store? As in clear a store completely

I have a similar issue perf when operating this service in production. Sometimes the GetCachedItemAsync operation takes 10 times the 90th percentile of the operation. I have 5 partitions, and each has 3 replicas. Sometimes it stays that way for a couple of minutes and causing timeout in my other services and my service's availability drops. And my service has HasPersistentState = false.

I have 2 suspects:

1. The distributedCacheLocator.GetCacheStoreProxy use a dictionary to cache the proxy, which never expires. Over time, the service fabric runtime might move the primary replica of the cache store to different node, it might cause a issue, or not? Not sure.

2. I suspected that if all the Get and Set operation is on one single replica( the primary replica) it will becomes a bottleneck. If other services on the primary replica is consuming CPU, it would slow down the operations.

   My scenario is: We have a lot of read operations, but write operations are not very frequent. So I figured that make the secondary replica available for read will ease this problem. I tried but failed because the GetCachedItemAsync, to my surprise, includes a SetCachedItemAsync call, which is a write and cannot be performed on secondary. And the GetCachedItemAsync is not virtual, so I cannot change this behavior. Is there any way to implement this? Or add an option? Or making GetCacheItemAsync virtual?

We have all of our remoting endpoints secured with certificates and need to provide an alternate implementation to the default one.

Is there a potential memory leak?

I am using service fabric distributed cache with a stateful service having HasPersistedState = true.
I believe HasPersistedState will allow me to have reliable collections backed by disk.
If I set MaxCacheSizeInMegabytes to 1000, does it mean all 1000 MB data is in memory backed by disk. Or does it mean Service fabric reliable collection will manage memory footprint having some objects in memory and rest all in disk.

Also, I see in the example HasPersistedState is false but replicationSize is set to 3. Why would we want replication if we are not backing the data to store there would be no way to recreate state in a node because of no snapshots?

Hi Team

Possible to run the service in service fabric + docker container?
Been trying for a couple of days and its not working.

The nuget artifact for this repository does not include the latest code changes, would you please publish the current state of this repository to the nuget package?

Hello!
Can I use this library in a Stateless Service fabric service? I did not manage to find this info in documentation.
Thank you in advance.

Hi,

is there any way to monitor the current cache size, number of entries in cache, reads/sec, writes/sec and so on? Ideally, via Azure Monitor metrics.

Thanks

While running load tests against our service, we're noticing that call to SetAsync() will sometimes fail with a TimeoutException, with what appears to be a 5 minute timeout.

System.TimeoutException: This can happen if message is dropped when service is busy or its long running operation and taking more time than configured Operation Timeout.

Tried setting the CancellationToken to limit the duration of the call, but it looks like that is not used in this method.

service-fabric-distributed-cache/src/SoCreate.Extensions.Caching.ServiceFabric/ServiceFabricDistributedCache.cs

Line 80 in bf2db07

await proxy.SetCachedItemAsync(key, value, options.SlidingExpiration, absoluteExpireTime).ConfigureAwait(false);

Our load tests did not seem particularly intensive. They only pushed the machine to around 20% CPU.

Redis is often used for Rate Limiting API calls, however this might be a great feature to add directly to this project?

For example consider implementing:

• SteppingTimeWindow
• SlidingTimeWindow
• LeakyBucket

Can it be added support to allow change settings of FabricTransportRemotingListenerSettings when the cache service is starting up, so that SecurityCredentials of this setting can be set accordingly to do a authenticated communication between cache client and cache service?

This is pretty important for serious applications to avoid unwanted client to talk to the cache service.

System.IO.FileNotFoundException: 'Could not load file or assembly 'C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\NS_7\Microsoft.ServiceFabric.Data.Impl.dll'. The system cannot find the path specified.'

Getting this exception in sample project and when i try to SOCreate distributed cache store in my project.

Library should be built for .net standard so that .net framework services may utilize it.

Hi all.

I'l create a PR for this if necessary, but right now package references are locked to specific patch version numbers. Can these not be 4.0.X?

You can imagine this creates an un necessary NuGet nightmare when using this cache package. Any chance of getting this updated?

Thanks!

The nuget artifact for this repository does not reflect its current state. Could you publish v3.2.0 for public use?

I'm trying to set the cache for a key. I think the basic code is there and working. When I run it without a CacheStoreServiceUri I get the following error.

Cache store not found in Service Fabric cluster. Try setting the 'CacheStoreServiceUri' configuration option to the location of your cache store.

When I set the CacheStoreServiceUri, I get the following error.

System.Fabric.FabricServiceNotFoundException: Service does not exist.

I think the problem is: I don't know what the CacheStoreServiceUri is supposed to be set to. The documentation says:

Used to explicitly point to the Stateful Reliable Service that is the cache store. If not supplied the client will try to auto discover the cache store.

The sample code has it as "fabric:/ServiceFabricDistributedCache/DistributedCacheStore". I took that and guessed what my URI might be. I don't think I guessed correctly.

Anyone got a suggestion on how to do this?

First - I really appreciate the work y'all have put in, I know maintaining this package isn't easy.

I believe this issue is separate from the timeout issue being discussed.

Our cache service encounters an irrevocable error around 6 hours after being deployed. We do have some variable sliding expiration times for items in the cache, however the expiration times don't line up exactly and the cache itself isn't actively being used. EX: We deploy and test the app that uses the cache at 4:50PM, with this exception being encountered at 11:02 PM. The cluster has the cache across 3 nodes, each of which fails at nearly identical times. It also appears that this error is occurring when building a replica of the service.

Looking at the source code, it appears to be line 204of DistributedCacheStoreService.

Other things to note: Data Loss was detected on these replicas seconds after recovery began, which makes sense if the replicas did not spin up.

Please let me know if there's any other way I can help or provide information.

Stack Trace Highlight:
Message: RunAsync failed due to an unhandled exception causing the host process to crash: System.ArgumentNullException: Value cannot be null. (Parameter 'key') at System.Fabric.Store.TStore'5.ConditionalGetAsync(IStoreTransaction transaction, TKey key, ReadMode readMode, TimeSpan timeout, CancellationToken cancellationToken) at System.Fabric.Store.TStore'5.GetAsync(IStoreTransaction transaction, TKey key, ReadMode readMode, TimeSpan timeout, CancellationToken cancellationToken) at Microsoft.ServiceFabric.Data.Collections.DistributedDictionary'2.TryGetValueAsync(Transaction tx, TKey key, LockMode lockMode, TimeSpan timeout, CancellationToken cancellationToken) at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.<>c__DisplayClass24_1.<<RemoveLeastRecentlyUsedCacheItemWhenOverMaxCacheSize>b__1>d.MoveNext()

Stack trace here:

Application: Secret.DistributedCache.exe CoreCLR Version: 4.700.19.56402 .NET Core Version: 3.1.0 Description: The application requested process termination through System.Environment.FailFast(string message). Message: RunAsync failed due to an unhandled exception causing the host process to crash: System.ArgumentNullException: Value cannot be null. (Parameter 'key') at System.Fabric.Store.TStore'5.ConditionalGetAsync(IStoreTransaction transaction, TKey key, ReadMode readMode, TimeSpan timeout, CancellationToken cancellationToken) at System.Fabric.Store.TStore'5.GetAsync(IStoreTransaction transaction, TKey key, ReadMode readMode, TimeSpan timeout, CancellationToken cancellationToken) at Microsoft.ServiceFabric.Data.Collections.DistributedDictionary'2.TryGetValueAsync(Transaction tx, TKey key, LockMode lockMode, TimeSpan timeout, CancellationToken cancellationToken) at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.<>c__DisplayClass24_1.<<RemoveLeastRecentlyUsedCacheItemWhenOverMaxCacheSize>b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.<>c__DisplayClass24_0.<<RemoveLeastRecentlyUsedCacheItemWhenOverMaxCacheSize>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at SoCreate.Extensions.Caching.ServiceFabric.RetryHelper.<>c__DisplayClass4_0.<<ExecuteWithRetry>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at SoCreate.Extensions.Caching.ServiceFabric.RetryHelper.ExecuteWithRetry[TResult](Func'3 operation, Object state, CancellationToken cancellationToken, Int32 maxAttempts, Nullable'1 initialDelay) at SoCreate.Extensions.Caching.ServiceFabric.RetryHelper.ExecuteWithRetry(IReliableStateManager stateManager, Func'4 operation, Object state, CancellationToken cancellationToken, Int32 maxAttempts, Nullable'1 initialDelay) at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.RemoveLeastRecentlyUsedCacheItemWhenOverMaxCacheSize(CancellationToken cancellationToken) at SoCreate.Extensions.Caching.ServiceFabric.DistributedCacheStoreService.RunAsync(CancellationToken cancellationToken) at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceReplicaAdapter.ExecuteRunAsync(CancellationToken runAsyncCancellationToken) Stack: at System.Environment.FailFast(System.String) at Microsoft.ServiceFabric.Services.Runtime.ServiceHelper+<>c__DisplayClass18_0.<HandleRunAsyncUnexpectedException>b__0() at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteFromThreadPool(System.Threading.Thread) at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()