[BUG] talos /var/lib/rancher/longhorn vs /var/lib/longhorn about longhorn HOT 6 CLOSED

It sounds like the setting default-data-path == /var/lib/rancher/longhorn in your environment. If so, longhorn-manager will try to create a default disk at that path. I think it's possible that additional adjustments will need to be made in the Talos environment if this is the case.

Can you check the setting? A support bundle may also be helpful here.

kubectl get setting -n longhorn-system default-data-path

cc @c3y1huang

from longhorn.

the thing is that the path is without rancher:

kubectl get setting -n longhorn-system default-data-path
NAME                VALUE               AGE
default-data-path   /var/lib/longhorn   3d19h

and my talos machine config also looks good:

    extraMounts:
      - destination: /var/lib/longhorn
        type: bind
        source: /var/lib/longhorn
        options:
          - bind
          - rshared
          - rw

I don't get why it is using the rancher path...

The support bundle was sent to the support email address.

from longhorn.

Looking at the support bundle, the default disk is indeed created with the path /var/lib/rancher/longhorn, but the setting is correctly /var/lib/longhorn. I will investigate it further.

from longhorn.

As I get started trying to understand where this /var/lib/rancher/longhorn string could have come from in the code itself, can you help to clarify a couple of things?

When analyzing the support bundle, I saw that all of the settings CRs were created on "2024-03-18T16:51:26Z". Some of them were immediately updated with the annotation longhorn.io/configmap-resource-version: "100269" at "2024-03-18T16:51:27Z". This probably indicates there was a longhorn-default-settings config map (created by Helm or kubectl) in the cluster at installation time. Only one setting, default-data-path, was updated more recently. It has longhorn.io/configmap-resource-version: "291113", which is the current resourceVersion of the longhorn-default-settings config map. The config map was updated at "2024-03-22T11:41:57Z". This indicates either the default-data-path setting was recently added to the longhorn-default-settings config map, or it was changed from a different value. Was this setting added to the config map or modified from a different value? It is important, because once the default disk for a node is created, it will never be created again. If the value was /var/lib/rancher/longhorn at installation time, a change to the setting will not fix the problem you are experiencing.

Since this cluster is freshly installed (and currently broken), can you try deleting the Longhorn node CRs with the settings as they currently are? If they come back with the same spec.disk.path, it seems there is no explanation other than Longhorn itself has produced the bad value.

kubectl delete -n longhorn-system nodes.longhorn.io --all

All node CRs currently have

      disks:
        default-disk-080600000000:
          allowScheduling: true
          diskType: filesystem
          evictionRequested: false
          path: /var/lib/rancher/longhorn
          storageReserved: 6.043572633e+09
          tags: []

from longhorn.

Thanks for you help that really helped me. I suspect during my initial flux install I might have had the old value in the config but I did not understand that once a bad value was provided all node objects use the old value... so changing anything in helm did not help. Remove the longhorn nodes solved it...

from longhorn.

That's good to hear @runningman84! And thanks for being quick on the update so I knew not to keep digging.

from longhorn.