[BUG] Git pillar lock files not being removed. about salt HOT 5 OPEN

The salt-master's configuration file only contains log_level_logfile: debug, all git fs settings are in raas, and this appears to be sseapi related. Will talk with raas team to investigate this further, as to how git fs is being used with regular open salt and from raas.

Chatted with the raas team and they don't use GitFS. Found the GitFS settings in a separate file masterXX_gitfs.conf which leverages azure

Q: how is azure accessed, are they using the azure salt extension ?

from salt.

Have a new master log file with the lock error's.
I am seeing a large number of Traceback errors in the file due to jinja2 rendering errors, this may be due to the pillar data is not accessible from GitFS, or some other issue, but investigating if the error paths on the GitFS access are not releasing the lock files, and that is why they still might be present (there may be an error path that was overlooked in the implementation).

from salt.

See probable cause, for example: process 50973 obtains a lock but never releases it, and gets killed, SIGTERM, and then another process attempts to obtain lock but finds process 509732 still has it, but it is dead. Note the long time difference between obtaining lock and SIGTERM, would have thought it would have been released soon after obtaining it, probable two bugs, should not hold locks that long, and SIGTERM should check and release resources held by a process like lock files.

Obtain lock

2024-01-11 09:20:26,218 [salt.utils.gitfs :999 ][DEBUG   ][50973] Set update lock for gitfs remote '[email protected]:v3/Vizientinc/Hosting/SaltStack'

SIGTERM approx. 16 hours later

2024-01-12 01:15:41,521 [salt.utils.process:1070][DEBUG   ][50973] FileserverUpdate received a SIGTERM. Exiting

Finding lock held by dead process

2024-01-12 01:16:25,647 [salt.utils.gitfs :961 ][WARNING ][3200] gitfs_global_lock is enabled and update lockfile /var/cache/salt/master/gitfs/work/NlJQs6Pss_07AugikCrmqfmqEFrfPbCDBqGLBiCd3oU=/_/update.lk is present for gitfs remote '[email protected]:v3/Vizientinc/Ho     sting/SaltStack'. Process 50973 obtained the lock but this process is not running. The update may have been interrupted. If using multi-master with shared gitfs cache, the lock may have been obtained by another master.

from salt.

The current testing for gitfs is only relying on _master_lock, and exercising it

_master_lock = multiprocessing.Lock()

and use acquire and release. this doesn't really test the locking in the GitProvider class with PyGit2 and GitPython lock & clear_lock mechanism but goes directly to underlying acquire and release,n should be testing the methods

from salt.

Have changes and tests in PR #65937 for handling GitFS lock files, and droppings from when the process is terminated, getting cleaned up.

from salt.