letsbuilders / jsonnet-libs Goto Github PK

We're forced to have a taint on spot instances on Azure clusters:

taints:
  - key: kubernetes.azure.com/scalesetpriority
    value: spot
    effect: NoSchedule

This means that if we want to run Pods on those the pods need to tolerate that taint. We want to run our Pods on spot instances as much as we can to save costs. Because our library doesn't currently support setting tolerations or affinities we add them to deployments by overriding them like here. This is a very crude approach and we should rather have it configurable with options defined in the _config object.

TODO:

• add support for configuring tolerations
• update all services that currently use overrides

We'd like to be able to easily configure pod affinities. We'd also like to prevent multiple Pods of the same deployment get scheduled onto the same node.

TODO:

• add support for configuring pod affinities
• add a default anti-affinity config that will suggest the scheduler not to put pods of the same deployment on one node.

Initially, HPA behavior was controlled by the existence of these fields:

• _config.deployment.autoscaling
• _config.statefulSet.autoscaling

but since #60 this field always exists because it contains default policies.
Due to that the replicas field is always hidden and if you want to set a fixed amount of replicas you need to "unhide" it, which is counter-intuitive.

job.mixin.spec.withTtlSecondsAfterFinished(180) which we add to Jobs doesn't work because the feature is hidden behind a feature gate that we don't have enabled.
EKS clusters don't have the option to enable feature gates on demand, so we'll most likely need to find different ways of cleaning up Jobs.

We use a far amount of custom implementations of Custom Resources for strimzi, crossplane, prometheus. We could alternatively use implementations generated by k8s-gen https://github.com/jsonnet-libs