Cannot use git lfs filter extensions to `clean` data which will not be `smudge`-d back about git-lfs HOT 3 OPEN

Quick addendum: I think I see why it needs the final oid guid in the pointer file - of course to find the file in lfs/objects, but also to know if the file has been modified. The intermediate guids I don't see a technical requirement for, except for the validation that currently happens when smudging.

With that in mind however, I do believe that it is not a functional requirement for the guids to be validated during a smudge process. It could be a flag defined in the config secion for a filter extension indicating whether it should do this validation or not, which has the downside that as soon as a filter has been run which does not want validation, any filters smudging the file afterwards could not do validation either even if they have the flag set requesting validation to be done for them.

from git-lfs.

Hey, thanks for writing up these notes!

it is unclear if low or high priority extensions run first. The common issue of priority being ascending and descending in different pieces of software.

I believe they are run in ascending priority order during a clean operation and run in descending priority order during a smudge operation. The documentation does seem to accord with that understanding. For the clean operation, it states:

If multiple extensions are installed, they are invoked in the order defined by their priority.

and for the smudge operation:

Each of the extensions indicated in the pointer file must be invoked in reverse order to undo the changes they made to the contents of the file.

I take your point that if an extension removes data destructively (such as a filter for sensitive content), it typically won't be able to reconstruct it when run "in reverse". I suppose that implies that any such filter must run only with the priority 1 so it runs first in the clean operation. Would that work for your experiment?

As for making the intermediate SHA-256 OIDs optional, I suspect that's not something we would add to our backlog at the moment, but if someone wants to contribute a patch with some tests which demonstrate how it works (and doesn't cause any incompatibilities with existing tools), we're always open to contributions!

from git-lfs.

I believe they are run in ascending priority order during a clean operation and run in descending priority order during a smudge operation. The documentation does seem to accord with that understanding. For the clean operation, it states:

If multiple extensions are installed, they are invoked in the order defined by their priority.

My bad for just mentioning it as a side note without explaining, that is exactly where I was confused. My thought was that adding "highest to lowest" or "lowest to highest" to the docs would make it much harder to misunderstand. Of course this varies from person to person, but to me I'd interpret the current wording as "highest priority first", which would be descending... or ascending depending on defintion 😅 . There's so much software using "priority" and "order" but how the implementations actually sort seems unpredictable. I hope that explains my confusion better. (But it's also not a huge deal to be clear.)

I take your point that if an extension removes data destructively (such as a filter for sensitive content), it typically won't be able to reconstruct it when run "in reverse". I suppose that implies that any such filter must run only with the priority 1 so it runs first in the clean operation. Would that work for your experiment?

Yea you got it. Though since I'm only using 1 extension here, its priority wouldn't make a difference. The ultimate issue in my case is that the final sha256 checksum of the file does get validated (the one calculated and saved before my extension ran, and then checked after the smudge for my extension ran). Making it run first would allow other extensions after it to be non destructive and therefore be able to get validated, but any destructive extensions are currently not possible so that detail technically does not matter (currently).

As for making the intermediate SHA-256 OIDs optional, I suspect that's not something we would add to our backlog at the moment, but if someone wants to contribute a patch with some tests which demonstrate how it works (and doesn't cause any incompatibilities with existing tools), we're always open to contributions!

Makes sense and no worries, I just wanted to give a bit of feedback. The workaround I mentioned - using a different git filter - appears to work quite nicely, so I'm sorry I won't be making such a PR (I've also not written a single line of go yet). But I'm glad that you would be open to such a contribution :)

from git-lfs.