[bitnami/ghost] Pods fail on init with GKE Autopilot and K8s Docker Desktop about charts HOT 5 CLOSED

Hi,

About this error

/opt/bitnami/scripts/libos.sh: line 346:    47 Killed                  "$@" > /dev/null 2>&1

It seems to me that the resource preset in your platform is not enough and requires a higher limit (which is strange as we test this chart in all major k8s distros), could you setting higher resource limits in the mysql.resources section? Another option could be mysql.resourcesPreset but this is not recommended in production.

from charts.

While GKE clusters are mostly likely on your major k8s distro list, perhaps the testing is on a GKE "standard" cluster, and the delineation of "GKE Autopilot" has not been attempted. My previous testing was without a sufficient understanding of the behaviors of ephemeral-storage as a resource with requests and limits.

Considering your suggestion I provided the chart with these override defaults, with ample storage.

# micro
resources:
  limits:
    cpu: 375m
    ephemeral-storage: 5Gi
    memory: 384Mi
  requests:
    cpu: 250m
    ephemeral-storage: 2Gi
    memory: 256Mi    

# micro
volumePermissions:
  resources:
    limits:
      cpu: 375m
      ephemeral-storage: 5Gi
      memory: 384Mi
    requests:
      cpu: 250m
      ephemeral-storage: 2Gi
      memory: 256Mi    

# small
mysql:
  primary:
    resources:
      limits:
        cpu: 700m
        ephemeral-storage: 5Gi
        memory: 768Mi
      requests:
        cpu: 500m
        ephemeral-storage: 2Gi
        memory: 512Mi 

The MySQL pod appears to be healthy and logging appropriately.

However, the ghost pod repeatedly tries to start and is evicted with this error:

Events:
  Type     Reason               Age   From                                   Message
  ----     ------               ----  ----                                   -------
  Normal   Scheduled            17m   gke.io/optimize-utilization-scheduler  Successfully assigned ghost/ghost-8fcc8fbf5-ztmdt to gk3-autopilot-cluster-1-pool-2-9a2f203f-hql9
  Normal   Pulled               17m   kubelet                                Container image "docker.io/bitnami/ghost:5.82.5-debian-12-r0" already present on machine
  Normal   Created              17m   kubelet                                Created container prepare-base-dir
  Normal   Started              17m   kubelet                                Started container prepare-base-dir
  Normal   Pulled               16m   kubelet                                Container image "docker.io/bitnami/ghost:5.82.5-debian-12-r0" already present on machine
  Normal   Created              16m   kubelet                                Created container ghost
  Normal   Started              16m   kubelet                                Started container ghost
  Warning  Evicted              16m   kubelet                                Pod ephemeral local storage usage exceeds the total limit of containers 50Mi.
  Normal   Killing              16m   kubelet                                Stopping container ghost
  Warning  ExceededGracePeriod  16m   kubelet                                Container runtime did not kill the pod within specified grace period.

Despite asking for 2Gi, there appears to be a hard limit on the GKE node which perhaps forces the Pods to have a 50Mi ceiling. This is evident as the Pod description reflects this 50Mi cap.:

    Limits:
      cpu:                375m
      ephemeral-storage:  50Mi
      memory:             384Mi
    Requests:
      cpu:                250m
      ephemeral-storage:  50Mi
      memory:             256Mi

My understanding is there are a few solutions:

1. Adjust the cluster's limit of ephemeral-storage. Since we cannot adjust the Autopilot criteria, the option would be to use a different cluster type GKE standard or another cluster flavor.

2. Change the ghost code so not as much data is stored to ephemeral-storage. I believe this also includes emptydir and any other files written to the container or pod that are not mounted to a PVC. For me and Bitnami chart authors, changing the Ghost code is not an immediate option. Perhaps there is a Ghost environment variable to reduce the verbosity of data being produced?

3. Mount the verbose output that exceeds the 50Mi limit to a PVC. At this point, I have not investigated what file(s) are getting so large and if they are going to emptyDir how to mount those to a PVC.

However, I'm confused because I was running a Helm chart for Ollama and was hitting the exact same eviction error related to ephemeral-storage. I increased the chart resource defaults with this:

resources:
  limits:
    ephemeral-storage: 5Gi
  requests:
    ephemeral-storage: 5Gi
    memory: 2Gi

and the problem went away and Ollama is healthy. With the Pod reporting the ephemeral-storage at 5Gi as requested:

   Requests:
      cpu:                308m
      ephemeral-storage:  5Gi
      memory:             2Gi

This tells me I do not understand Autopilot or assume that's what limits the ephemeral-storage request. Why would the Bitnami Ghost deployment be capped at 50Mi, yet this other chart has no problem requesting 5Gi?

Thoughts?

from charts.

Hi,

This is weird, as we do not set any kind of cap in the resources. Could you confirm that the rendered YAML indeed sets the ephemeral-storage to 5Gi? As this 50Mi could be related to the resources section not being rendered correctly

from charts.

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

from charts.

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

from charts.