cloudworkz / cortex-ops Goto Github PK

Memcache can have a great positive impact on performance. If setup incorrectly you won't benefit much from it. It's not easy to tell whether you are properly utilizing memcache. For instance:

1. Is your Memcache cluster oversized / undersized?
2. Should you reserve some memcache storage for the different components?
   a) Is it more important to cache Query Frontend stuff than index writes?
   b) If yes, does it make sense to have multiple memcache clusters, each component has it's own to make sure they have dedicated storage sizes? Or is it possible to configure memcache so that each component can have it's own dedicated storage within a single cluster

Slack discussion: https://cloud-native.slack.com/archives/CCYDASBLP/p1574096262229600

To ensure a healthy number of ingester replicas one needs pod disruption budgets. Even though the hard anti affinity on nodes helps, it can not guarantee to have more than n - i healthy ingesters, where n is the number of ingester replicas and i the number of metric replicas configured in the distributor.

Ref: hashicorp/terraform-provider-kubernetes#320

Once available in Terraform we should add a podsecuritypolicy which will be eused by all components. Waiting for hashicorp/terraform-provider-kubernetes#374 to be resolved.

Hi

I was having a look at your repo as an example and I bumped into this. I though it might be worthy to mention, just to avoid any possible headache it might cause :)

We should investigate and clarify whether and under what circumstances it makes sense to enable or disable the consistent hashing feature for Memcache. From the memcache docs:

Consistent Hashing is a model that allows for more stable distribution of keys given addition or removal of servers. [...]

https://github.com/memcached/memcached/wiki/ConfiguringClient#consistent-hashing

We have flags to enable it:

-memcached.consistent-hash=true
-store.index-cache-read.memcached.consistent-hash=true

I am not sure on what cortex components this should be enabled and how the migration from non consistent hashing to consistent hashing works. Can we just switch this without worrying about that, or should we clear the cache after enabling this?

In order to be more open to the community we should use Kubernetes manifests for the deployment rather than Terraform modules. This way we can ensure everyone understands the deployment configs and therefore can also contribute improvements.

On top of that we don't need to wait for the Terraform Kubernetes provider to catch up with lacking features.