Pull doesn't work from v0.39.1 since fetchAll is enabled by default about lazygit HOT 33 CLOSED

What do the logs say? Run lazygit --debug in one terminal and lazygit --logs in another, and then try pulling.

Does this happen with all repos or just this one?

Does pulling from the cli work?

from lazygit.

Sounds related to #2999, maybe. That, however, should be no different between 0.38.2 and 0.40.2.

The most important question would be if this happens always, or only sometimes. If it's only sometimes, then my guess would be that it's related to the sync mutex (like #2999), and in that case it would be solved by #3021, which removes the sync mutex. @Tim-Cao Are you able to build from source and could try a master build?

from lazygit.

Hi @mark2185
lazygit --debug

POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 54 lock 0xc0004061b0
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:51 oscommands.(*cmdObjRunner).RunWithOutput ??? <<<<<
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:50 oscommands.(*cmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:34 commands.(*gitCmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 commands.(*gitCmdObjRunner).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:189 oscommands.(*CmdObj).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:77 git_commands.(*SyncCommands).FetchBackground ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:107 gui.(*BackgroundRoutineMgr).backgroundFetch ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:59 gui.(*BackgroundRoutineMgr).startBackgroundFetch ???
/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:96 utils.Safe.func1 ???
/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:109 utils.SafeWithError ???
/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:97 utils.Safe ???

Have been trying to lock it again for more than 30s
goroutine 74 lock 0xc0004061b0
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:51 oscommands.(*cmdObjRunner).RunWithOutput ??? <<<<<
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:50 oscommands.(*cmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:34 commands.(*gitCmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 commands.(*gitCmdObjRunner).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:189 oscommands.(*CmdObj).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:98 git_commands.(*SyncCommands).Pull ???
/home/runner/work/lazygit/lazygit/pkg/gui/controllers/sync_controller.go:150 controllers.(*SyncController).pullWithLock ???
/home/runner/work/lazygit/lazygit/pkg/gui/controllers/sync_controller.go:143 controllers.(*SyncController).PullAux.func1 ???
/home/runner/work/lazygit/lazygit/pkg/gui/popup/popup_handler.go:155 popup.(*PopupHandler).WithLoaderPanel.func1 ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:679 gocui.(*Gui).onWorkerAux ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:665 gocui.(*Gui).OnWorker.func1 ???

Here is what goroutine 54 doing now
goroutine 54 [syscall]:
syscall.Syscall6(0xc00068b828?, 0x4eb0ef?, 0x7f9f21e61868?, 0xc00068b8d0?, 0x415d30?, 0xc00068b880?, 0x4fa55e?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:91 +0x36
os.(*Process).blockUntilWaitable(0xc0002e4000)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/wait_waitid.go:32 +0x87
os.(*Process).wait(0xc0002e4000)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/exec_unix.go:22 +0x28
os.(*Process).Wait(...)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000198160)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/exec/exec.go:890 +0x45
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runAndStreamAux(0xc000408560, {0xdeec88, 0xc0001f6040}, 0xc00068bca8)
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:255 +0x68c
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runAndDetectCredentialRequest(0xc000408560, {0xdeec88, 0xc0001f6040}, 0xd23e38)
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:328 +0xc8
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runWithCredentialHandling(0xc0001d8d78?, {0xdeec88, 0xc0001f6040})
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:303 +0x54
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).RunWithOutput(0xc0001f6000?, {0xdeec88, 0xc0001f6040})
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:56 +0x12c
github.com/jesseduffield/lazygit/pkg/commands.(*gitCmdObjRunner).RunWithOutput(0xc000418468, {0xdeec88, 0xc0001f6000})
	/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:33 +0xe7
github.com/jesseduffield/lazygit/pkg/commands.(*gitCmdObjRunner).Run(0xc000404000?, {0xdeec88?, 0xc0001f6000?})
	/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 +0x25
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*CmdObj).Run(0xc000422158?)
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:190 +0x2f
github.com/jesseduffield/lazygit/pkg/commands/git_commands.(*SyncCommands).FetchBackground(0xc00038f040?)
	/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:78 +0x22
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).backgroundFetch(0xc000408570)
	/home/runner/work/lazygit/lazygit/pkg/gui/background.go:104 +0x30
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).startBackgroundFetch(0xc000408570)
	/home/runner/work/lazygit/lazygit/pkg/gui/background.go:59 +0x71
github.com/jesseduffield/lazygit/pkg/utils.Safe.func1()
	/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:97 +0x1b
github.com/jesseduffield/lazygit/pkg/utils.SafeWithError(0x0?)
	/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:108 +0x67
github.com/jesseduffield/lazygit/pkg/utils.Safe(0x0?)
	/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:97 +0x35
created by github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).startBackgroundRoutines
	/home/runner/work/lazygit/lazygit/pkg/gui/background.go:31 +0xdb

lazygit --logs

Dec 26 06:46:03 |INFO| postRefreshUpdate for files took 4.855µs 
Dec 26 06:46:04 |INFO| refreshing the following scopes in sync mode: files 
Dec 26 06:46:04 |INFO| refreshed merge conflicts in 20.086µs 
Dec 26 06:46:04 |DEBU| using cache for key status.showUntrackedFiles 
Dec 26 06:46:04 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Dec 26 06:46:05 |INFO| git status --untracked-files=all --porcelain -z (924.575047ms) 
Dec 26 06:46:05 |INFO| refreshed files in 925.653063ms 
Dec 26 06:46:05 |INFO| Refresh took 925.940836ms 
Dec 26 06:46:05 |INFO| postRefreshUpdate for submodules took 7.776µs 
Dec 26 06:46:05 |INFO| postRefreshUpdate for files took 5.681µs 
Dec 26 06:46:14 |INFO| refreshing the following scopes in sync mode: files 
Dec 26 06:46:14 |INFO| refreshed merge conflicts in 5.881µs 
Dec 26 06:46:14 |DEBU| using cache for key status.showUntrackedFiles 
Dec 26 06:46:14 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Dec 26 06:46:15 |INFO| git status --untracked-files=all --porcelain -z (926.979441ms) 
Dec 26 06:46:15 |INFO| refreshed files in 927.226045ms 
Dec 26 06:46:15 |INFO| Refresh took 927.38407ms 
Dec 26 06:46:15 |INFO| postRefreshUpdate for submodules took 11.135µs 
Dec 26 06:46:15 |INFO| postRefreshUpdate for files took 8.769µs 

It happens on all repos and pulling from cli works

from lazygit.

Hi @stefanhaller, master build also doesn't work. It happens always.

from lazygit.

The panic log you posted above clearly shows the sync lock problem that I was referring to above. It would be interesting to see a similar log from running master, which doesn't have the sync lock any more.

If you can reliably reproduce the issue with v0.40.2, but not with v0.38.2, then it would be most helpful if you could bisect to the commit where it starts to fail for you.

from lazygit.

Hi @stefanhaller, this issue starts with v0.39.1. The root cause should be #2692
If I add fetchAll: false to config, the pull works as expected with v0.38.2.
Would you mind having a look at if fetchAll: false should be by default? Thanks

from lazygit.

Thanks for investigating. It seems you have one remote that hangs when you try to fetch it. Does git fetch --all on the command line work for you?

As for the config default, we had an elaborate discussion about this in #2692, I don't see a reason to question the decision right now.

from lazygit.

hmm, actually, I have dozens of remotes. But I only need to sync to upstream or origin selected by master branch. It wastes too much time to fetch all.

from lazygit.

hmm, actually, I have dozens of remotes. But I only need to sync to upstream or origin selected by master branch. It wastes too much time to fetch all.

So are you saying that it doesn't actually hang, but only takes a long time? In that case it sounds like you just need to set the fetchAll config to false because that's your preference, and we don't have an issue here.

from lazygit.

Hi @stefanhaller,
Is there any benefit to invoke fetchAll when run pulling?

from lazygit.

Hi @stefanhaller, Is there any benefit to invoke fetchAll when run pulling?

No, I don't think so. But I don't understand why you are asking; lazygit doesn't pass --all when pulling, even if the fetchAll config is true. The fetchAll config only affects the periodic background fetch, and fetching manually by pressing f in the files panel.

from lazygit.

However, if the fetchAll is true, when run pulling in Branches panel, it actually triggered fetch --all

While the fetchAll is false, it invokes the git pull --no-edit.

from lazygit.

I don't think so. My guess is that the "Fetching" output is from a background fetch. I'm a little bit confused about the screenshot because the output from two commands seem to be intermingled, but my guess is that this is not a master build, but the latest release, which still has the sync lock. So what happens is that right after startup the first background fetch starts, grabs the sync lock, prints the "Fetching" lines, and then hangs for some reason. Then, when you pull, it prints "Pull" to the log, and before doing anything it blocks forever trying to get the sync lock.

So again, two questions that I asked before, but didn't get an answer for:

• when you type git fetch --all on the command line, does this hang? (If so, you might want to try to find the remote that hangs, and remove it or fix its URL.)
• you said above that a master build still has the same problem. I find this hard to believe as there's no sync lock any more on master. Could you please send debug logs from a master build?

from lazygit.

Hi @stefanhaller, I just tested on master build again. I'm at d97b37a
It requires pull.ff.only settings, is that expected changes?

from lazygit.

This warning doesn't come from lazygit, but from git itself. I don't understand why you are getting the warning here; usually it only appears when your local branch diverged from upstream, which doesn't seem to be the case here (master can be fast-forwarded). I don't have any ideas about this one, anybody else?

In general, btw, my advice is to set git config --global pull.rebase true.

from lazygit.

Hi @stefanhaller, the warning might be a bug on master of lazygit, if doesn't set up pull configuration, that warning will always show in the first time but disappear in the second time. In previous published versions, it doesn't have that warning in the first time.
Would you mind having a look at? Also, the fetch -all still be tiggered if doesn't have the fetchAll false

from lazygit.

Thanks for the video. It looks like the first pull happens at the same time as the initial background fetch when it's still busy fetching master's upstream; maybe there's some concurrency issue related to this. We already know that there can be cases where spurious errors occur because of this (see for example #3180 (comment), which is slightly different). I don't totally understand what's going on yet, but I can try to look into it more.

Meanwhile you probably want to set fetchAll to false, as you said above. Your video shows that fetching your remotes is so slow that you probably don't want to do that all the time.

from lazygit.

Hi @stefanhaller,

maybe there's some concurrency issue related to this. We already know that there can be cases where spurious errors occur because of this (see for example #3180 (comment), which is slightly different). I don't totally understand what's going on yet, but I can try to look into it more.

They are same. If have pull config, like git config --global pull.rebase true, the warning message becomes to cannot rebase into multiple branches

Meanwhile you probably want to set fetchAll to false, as you said above. Your video shows that fetching your remotes is so slow that you probably don't want to do that all the time.

Exactly! Thanks

from lazygit.

Is there any update? @stefanhaller

from lazygit.

Yes. I just opened a PR (#3202) that fixes the "Cannot rebase onto multiple branches" error. I couldn't reproduce the other error ("You have divergent branches") locally, so I'm not sure if it also fixes that one. Would be great if you could test this, if you can build from source.

from lazygit.

Hi @stefanhaller, I tested on the #3202, both Cannot rebase onto multiple branches and You have divergent branches are fixed.

But this issue still happens, do you have another pr to fix it? Additionally, if quit the lazygit during fetching, it will continue when launch lazygit again.

The fetch -all still be tiggered if doesn't have the fetchAll false
Fetching your remotes is so slow that you probably don't want to do that all the time.

from lazygit.

Hi @stefanhaller, I tested on the #3202, both Cannot rebase onto multiple branches and You have divergent branches are fixed.

That's good to hear, thanks for testing.

But this issue still happens, do you have another pr to fix it? Additionally, if quit the lazygit during fetching, it will continue when launch lazygit again.

The fetch -all still be tiggered if doesn't have the fetchAll false
Fetching your remotes is so slow that you probably don't want to do that all the time.

I don't understand the problem. This is by design, fetch --all is the default, and you can turn it off by setting the fetchAll config to false if you don't want it. What's the problem?

from lazygit.

Hi @stefanhaller,
I don't understand why did this change that when run pulling, it will trigger both git pull --no-edit and git fetch -all instead of just git pull --no-edit like before?

from lazygit.

Hi @stefanhaller, I don't understand why did this change that when run pulling, it will trigger both git pull --no-edit and git fetch -all instead of just git pull --no-edit like before?

That's not what it's doing. When you do a pull, it does not call fetch, only pull.

What you are seeing is the initial background fetch; in the v0.40.2 release the pull command would block until the background fetch is finished, but now on master the pull is allowed to run in parallel to the background fetch. That's why you see both next to each other in the log. You can confirm this by waiting for the fetch to be complete, and then pressing "p"; you'll see that it doesn't call fetch.

from lazygit.

Hi @stefanhaller,
Would you mind explaining what does the initial background fetch for?

from lazygit.

Lazygit has a feature to periodically run git fetch in the background, to keep your remotes up to date without you having to fetch manually. The first of these runs right after startup, from then on it runs every 60 seconds by default. You can change the frequency with the refresher.fetchInterval option, and you can turn it off altogether by setting git.autoFetch to false (it's on by default). See Config.md for more about lazygit's config options.

from lazygit.

@stefanhaller
I have 41 remotes, Should I turn it off by default?

from lazygit.

That's your choice. I thought we agreed above that it makes sense for you to turn off git.fetchAll, since fetching is so slow for you. Whether you also (or instead) want to turn off git.autoFetch is up to you.

from lazygit.

@stefanhaller
I tried if disable git.autoFetch on master build, then don't have to disable git.fetchAll, and vice versa. Is that expected?

from lazygit.

No, fetchAll not only applies to the background fetch, but also to pressing "f" in the files panel to fetch manually, so you still might want to decide for both whether you want them on or off.

from lazygit.

@stefanhaller
Do you mean disable git.autoFetch is enough in my case, right?

from lazygit.

That would solve it for you, yes. Whether that's what you want is not something I can decide for you; you'll have to decide this yourself.

from lazygit.

Got it. I really appreciate your help. Look forward to the new release. @stefanhaller

from lazygit.