[BACKPORT][v1.5.5][BUG] Backup marked as "completed" cannot be restored, gzip: invalid header about longhorn HOT 3 CLOSED

Verifed pass on longhorn v1.5.x (longhorn-engine 2f9588) with test cases

Test case 1 - Pass
Test case 2 - Pass
Test case 3 - N/A, v1.5.x did not support backup backing image
Test case 4 - restore and backup test cases did not see outstanging issue

• https://ci.longhorn.io/job/private/job/longhorn-tests-regression/6728/
• https://ci.longhorn.io/job/private/job/longhorn-tests-regression/6729/ (failed case rerun passed at #6732, #6733)

from longhorn.

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

We are still investigating the root cause of the issue.
The PRs improve the resilience in the decompression path.

• Test case 1

  1. Create a volume in LH v1.4.x, write data and take backup multiple times
  2. Update to v1.5.x, restore the backups. The restoration should succeed.

• Test case 2

  1. Create a volume in LH v1.5.x, write data and take backup multiple times
  2. Restore the backups. The restoration should succeed.

• Test case 3 (cc @ChanYiLin )

  1. Back up backing images and restore it

• Test case 4

  1. The test cases in e2e related to backup/restore should pass

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

longhorn/backupstore#167
longhorn/longhorn-engine#1076

• Which areas/issues this PR might have potential impacts on?
  Area: backup and restore
  Issues

from longhorn.

Thank you @chriscchien
cc @innobead

from longhorn.