[BUG] no Pending workload pods for volume xxx to be mounted about longhorn HOT 7 OPEN

I can reproduce the issue by the following steps

1. Create a LH v1.6.0 2-nodes cluster
2. Deploy a deployment
3. Restart kubelet on the attached node (systemctl restart k3s-agent.service if bootstrapped by k3s)
4. Then, the no Pending workload pods for volume xxx to be mounted message is repeatedly emitted

cc @c3y1huang

from longhorn.

@c3y1huang Would you be able to take a look? Thanks.

Ref: https://github.com/longhorn/longhorn-manager/pull/2494/files

from longhorn.

@c3y1huang Please also add this issue to the known issue wiki page for the impacted versions.

from longhorn.

Can you provide a support bundle for analysis?

from longhorn.

I sent it to [email protected] this morning (I suspect it contains identifying information).

from longhorn.

@Pionerd

1. Seems like somehow Kubelet dispatched another NodePublishVolume despite the Pod being already in the running state. Could you verify that were the state of those pods?
2. I cannot find related events in the support-bundle, I guess because they get cleared out after an hour. Is the no Pending workload pods for volume xxx to be mounted recurring?

from longhorn.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: #8072 (comment)

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at: issue description

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at longhorn/longhorn-manager#2659

• Which areas/issues this PR might have potential impacts on?
  Area CSI
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at longhorn/longhorn-tests#1796

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

from longhorn.