[BACKPORT][v1.6.1][BUG] Longhorn api-server PUT request rate about longhorn HOT 7 CLOSED

Hello @roger-ryao, can you please confirm that you also ran longhorn/longhorn-manager#2685 (comment) to test this fix? Sorry for the confusion. I do think longhorn/longhorn-manager#2432 (comment) should be run again (as you have done), but the former are the primary test steps.

Moving this back to Ready for Testing. Please feel free to close it again directly if you have already run these steps.

from longhorn.

This is correct. The necessary changes did not make it to longhorn-manager in https://github.com/longhorn/longhorn-manager/commits/v1.6.1-rc1. They are only in https://github.com/longhorn/longhorn-manager/commits/v1.6.x/.

cc @roger-ryao

from longhorn.

ref: #8167

cc @ejweber

from longhorn.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: longhorn/longhorn-manager#2685 (comment).
  Run the following test again: longhorn/longhorn-manager#2432 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  @krohrsb managed to mitigate it by increasing the capability of his API server (#8114 (comment)), but there is nothing a user can do to tweak Longhorn to avoid the issue.

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: #8167.
  The PR for the chart change is at: #8167.

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: longhorn/longhorn-manager#2694.

from longhorn.

Verified on v1.6.1-rc1 20240314

• longhorn v1.6.1-rc1 6e96e27

The test steps
longhorn/longhorn-manager#2432 (comment)
#7425 (comment)

Result Passed

1. If you wait any length of time, ReplicaA is never cleaned up.
2. After deleting ReplicaB and waiting for the volume to rebuild from ReplicaA, the data is consistent.

from longhorn.

Verified on v1.6.1-rc1 20240318

• longhorn v1.6.1-rc1 6e96e27

The test steps
longhorn/longhorn-manager#2685 (comment)
There are four cases to verify:

1. Ensure that the upgrade correctly populates replicaTransitionTimeMap
2. Confirm that the rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica deletion
3. Validate that the rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica failure
4. Ensure that replicas are not updated unnecessarily when the cluster is stable.

Result

1. We observed that the replicaTransitionTimeMap is not visible on v1.6.1-rc1, but the test passed on the master-head.
2. Attaching the screenshot, PUT requests to replica resources persist. Additionally, after updating Longhorn from v1.6.1-rc1 to the master-head, no PUT requests to replica resources were observed.

cc. @ejweber

from longhorn.

Verified on v1.6.x-head 20240319

• longhorn v1.6.x-head 6e96e27
• longhorn-manager v1.6.x-head https://github.com/longhorn//longhorn-manager/commit/0dd766f7376b13bc6ecc1c94f6b7a311d72e84e8

Result Passed

• 1. We observed that the replicaTransitionTimeMap is visible on v1.6.x-head.
• 2. The rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica deletion
• 3. The rebuild operation correctly populates replicaTransitionTimeMap and lastHealthyAt after replica failure
• 4. After updating Longhorn from v1.6.1-rc1 to the v1.6.x-head, no PUT requests to replica resources were observed.

from longhorn.