Pull doesn't work from v0.39.1 since fetchAll is enabled by default #3181

Tim-Cao · 2023-12-22T19:17:35Z

Describe the bug
Pull doesn't work on v0.40.2

To Reproduce
Steps to reproduce the behavior:

Go to branches panel
Press p

Expected behavior
The pull should complete less than 1 mins.

Actual behavior
The pull keeps running.

Screenshots

Version info:
commit=5e388e21c8ca6aa883dbcbe45c47f6fdd5116815, build date=2023-08-07T14:05:48Z, build source=binaryRelease, version=0.40.2, os=linux, arch=amd64, git version=2.43.0
ubuntu 22.04

Last Working Version
v0.38.2

mark2185 · 2023-12-23T08:37:47Z

What do the logs say? Run lazygit --debug in one terminal and lazygit --logs in another, and then try pulling.

Does this happen with all repos or just this one?

Does pulling from the cli work?

stefanhaller · 2023-12-23T17:51:05Z

Sounds related to #2999, maybe. That, however, should be no different between 0.38.2 and 0.40.2.

The most important question would be if this happens always, or only sometimes. If it's only sometimes, then my guess would be that it's related to the sync mutex (like #2999), and in that case it would be solved by #3021, which removes the sync mutex. @Tim-Cao Are you able to build from source and could try a master build?

Tim-Cao · 2023-12-25T22:49:31Z

Hi @mark2185
lazygit --debug

POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 54 lock 0xc0004061b0
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:51 oscommands.(*cmdObjRunner).RunWithOutput ??? <<<<<
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:50 oscommands.(*cmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:34 commands.(*gitCmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 commands.(*gitCmdObjRunner).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:189 oscommands.(*CmdObj).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:77 git_commands.(*SyncCommands).FetchBackground ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:107 gui.(*BackgroundRoutineMgr).backgroundFetch ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:59 gui.(*BackgroundRoutineMgr).startBackgroundFetch ???
/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:96 utils.Safe.func1 ???
/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:109 utils.SafeWithError ???
/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:97 utils.Safe ???

Have been trying to lock it again for more than 30s
goroutine 74 lock 0xc0004061b0
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:51 oscommands.(*cmdObjRunner).RunWithOutput ??? <<<<<
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:50 oscommands.(*cmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:34 commands.(*gitCmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 commands.(*gitCmdObjRunner).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:189 oscommands.(*CmdObj).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:98 git_commands.(*SyncCommands).Pull ???
/home/runner/work/lazygit/lazygit/pkg/gui/controllers/sync_controller.go:150 controllers.(*SyncController).pullWithLock ???
/home/runner/work/lazygit/lazygit/pkg/gui/controllers/sync_controller.go:143 controllers.(*SyncController).PullAux.func1 ???
/home/runner/work/lazygit/lazygit/pkg/gui/popup/popup_handler.go:155 popup.(*PopupHandler).WithLoaderPanel.func1 ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:679 gocui.(*Gui).onWorkerAux ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:665 gocui.(*Gui).OnWorker.func1 ???

Here is what goroutine 54 doing now
goroutine 54 [syscall]:
syscall.Syscall6(0xc00068b828?, 0x4eb0ef?, 0x7f9f21e61868?, 0xc00068b8d0?, 0x415d30?, 0xc00068b880?, 0x4fa55e?)
	/opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_linux.go:91 +0x36
os.(*Process).blockUntilWaitable(0xc0002e4000)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/wait_waitid.go:32 +0x87
os.(*Process).wait(0xc0002e4000)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/exec_unix.go:22 +0x28
os.(*Process).Wait(...)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc000198160)
	/opt/hostedtoolcache/go/1.20.7/x64/src/os/exec/exec.go:890 +0x45
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runAndStreamAux(0xc000408560, {0xdeec88, 0xc0001f6040}, 0xc00068bca8)
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:255 +0x68c
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runAndDetectCredentialRequest(0xc000408560, {0xdeec88, 0xc0001f6040}, 0xd23e38)
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:328 +0xc8
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runWithCredentialHandling(0xc0001d8d78?, {0xdeec88, 0xc0001f6040})
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:303 +0x54
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).RunWithOutput(0xc0001f6000?, {0xdeec88, 0xc0001f6040})
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:56 +0x12c
github.com/jesseduffield/lazygit/pkg/commands.(*gitCmdObjRunner).RunWithOutput(0xc000418468, {0xdeec88, 0xc0001f6000})
	/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:33 +0xe7
github.com/jesseduffield/lazygit/pkg/commands.(*gitCmdObjRunner).Run(0xc000404000?, {0xdeec88?, 0xc0001f6000?})
	/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 +0x25
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*CmdObj).Run(0xc000422158?)
	/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:190 +0x2f
github.com/jesseduffield/lazygit/pkg/commands/git_commands.(*SyncCommands).FetchBackground(0xc00038f040?)
	/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:78 +0x22
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).backgroundFetch(0xc000408570)
	/home/runner/work/lazygit/lazygit/pkg/gui/background.go:104 +0x30
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).startBackgroundFetch(0xc000408570)
	/home/runner/work/lazygit/lazygit/pkg/gui/background.go:59 +0x71
github.com/jesseduffield/lazygit/pkg/utils.Safe.func1()
	/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:97 +0x1b
github.com/jesseduffield/lazygit/pkg/utils.SafeWithError(0x0?)
	/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:108 +0x67
github.com/jesseduffield/lazygit/pkg/utils.Safe(0x0?)
	/home/runner/work/lazygit/lazygit/pkg/utils/utils.go:97 +0x35
created by github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).startBackgroundRoutines
	/home/runner/work/lazygit/lazygit/pkg/gui/background.go:31 +0xdb

lazygit --logs

Dec 26 06:46:03 |INFO| postRefreshUpdate for files took 4.855µs 
Dec 26 06:46:04 |INFO| refreshing the following scopes in sync mode: files 
Dec 26 06:46:04 |INFO| refreshed merge conflicts in 20.086µs 
Dec 26 06:46:04 |DEBU| using cache for key status.showUntrackedFiles 
Dec 26 06:46:04 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Dec 26 06:46:05 |INFO| git status --untracked-files=all --porcelain -z (924.575047ms) 
Dec 26 06:46:05 |INFO| refreshed files in 925.653063ms 
Dec 26 06:46:05 |INFO| Refresh took 925.940836ms 
Dec 26 06:46:05 |INFO| postRefreshUpdate for submodules took 7.776µs 
Dec 26 06:46:05 |INFO| postRefreshUpdate for files took 5.681µs 
Dec 26 06:46:14 |INFO| refreshing the following scopes in sync mode: files 
Dec 26 06:46:14 |INFO| refreshed merge conflicts in 5.881µs 
Dec 26 06:46:14 |DEBU| using cache for key status.showUntrackedFiles 
Dec 26 06:46:14 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Dec 26 06:46:15 |INFO| git status --untracked-files=all --porcelain -z (926.979441ms) 
Dec 26 06:46:15 |INFO| refreshed files in 927.226045ms 
Dec 26 06:46:15 |INFO| Refresh took 927.38407ms 
Dec 26 06:46:15 |INFO| postRefreshUpdate for submodules took 11.135µs 
Dec 26 06:46:15 |INFO| postRefreshUpdate for files took 8.769µs

It happens on all repos and pulling from cli works

Tim-Cao · 2023-12-25T23:08:56Z

Hi @stefanhaller, master build also doesn't work. It happens always.

stefanhaller · 2023-12-26T13:09:33Z

The panic log you posted above clearly shows the sync lock problem that I was referring to above. It would be interesting to see a similar log from running master, which doesn't have the sync lock any more.

If you can reliably reproduce the issue with v0.40.2, but not with v0.38.2, then it would be most helpful if you could bisect to the commit where it starts to fail for you.

Tim-Cao · 2023-12-26T23:57:40Z

Hi @stefanhaller, this issue starts with v0.39.1. The root cause should be #2692
If I add fetchAll: false to config, the pull works as expected with v0.38.2.
Would you mind having a look at if fetchAll: false should be by default? Thanks

stefanhaller · 2023-12-27T07:42:40Z

Thanks for investigating. It seems you have one remote that hangs when you try to fetch it. Does git fetch --all on the command line work for you?

As for the config default, we had an elaborate discussion about this in #2692, I don't see a reason to question the decision right now.

Tim-Cao · 2023-12-27T23:29:05Z

hmm, actually, I have dozens of remotes. But I only need to sync to upstream or origin selected by master branch. It wastes too much time to fetch all.

stefanhaller · 2023-12-28T07:31:58Z

hmm, actually, I have dozens of remotes. But I only need to sync to upstream or origin selected by master branch. It wastes too much time to fetch all.

So are you saying that it doesn't actually hang, but only takes a long time? In that case it sounds like you just need to set the fetchAll config to false because that's your preference, and we don't have an issue here.

Tim-Cao · 2023-12-28T20:43:03Z

Hi @stefanhaller,
Is there any benefit to invoke fetchAll when run pulling?

stefanhaller · 2023-12-29T11:38:53Z

Hi @stefanhaller, Is there any benefit to invoke fetchAll when run pulling?

No, I don't think so. But I don't understand why you are asking; lazygit doesn't pass --all when pulling, even if the fetchAll config is true. The fetchAll config only affects the periodic background fetch, and fetching manually by pressing f in the files panel.

Tim-Cao · 2023-12-29T13:16:18Z

However, if the fetchAll is true, when run pulling in Branches panel, it actually triggered fetch --all

While the fetchAll is false, it invokes the git pull --no-edit.

stefanhaller · 2023-12-29T13:40:27Z

I don't think so. My guess is that the "Fetching" output is from a background fetch. I'm a little bit confused about the screenshot because the output from two commands seem to be intermingled, but my guess is that this is not a master build, but the latest release, which still has the sync lock. So what happens is that right after startup the first background fetch starts, grabs the sync lock, prints the "Fetching" lines, and then hangs for some reason. Then, when you pull, it prints "Pull" to the log, and before doing anything it blocks forever trying to get the sync lock.

So again, two questions that I asked before, but didn't get an answer for:

when you type git fetch --all on the command line, does this hang? (If so, you might want to try to find the remote that hangs, and remove it or fix its URL.)
you said above that a master build still has the same problem. I find this hard to believe as there's no sync lock any more on master. Could you please send debug logs from a master build?

Tim-Cao · 2023-12-29T23:17:16Z

Hi @stefanhaller, I just tested on master build again. I'm at d97b37a
It requires pull.ff.only settings, is that expected changes?

stefanhaller · 2023-12-30T14:32:11Z

This warning doesn't come from lazygit, but from git itself. I don't understand why you are getting the warning here; usually it only appears when your local branch diverged from upstream, which doesn't seem to be the case here (master can be fast-forwarded). I don't have any ideas about this one, anybody else?

In general, btw, my advice is to set git config --global pull.rebase true.

Tim-Cao · 2023-12-31T00:15:01Z

Hi @stefanhaller, the warning might be a bug on master of lazygit, if doesn't set up pull configuration, that warning will always show in the first time but disappear in the second time. In previous published versions, it doesn't have that warning in the first time.
Would you mind having a look at? Also, the fetch -all still be tiggered if doesn't have the fetchAll false

Actual.mp4

stefanhaller · 2023-12-31T12:22:04Z

Thanks for the video. It looks like the first pull happens at the same time as the initial background fetch when it's still busy fetching master's upstream; maybe there's some concurrency issue related to this. We already know that there can be cases where spurious errors occur because of this (see for example #3180 (comment), which is slightly different). I don't totally understand what's going on yet, but I can try to look into it more.

Meanwhile you probably want to set fetchAll to false, as you said above. Your video shows that fetching your remotes is so slow that you probably don't want to do that all the time.

Tim-Cao · 2024-01-01T11:32:59Z

Hi @stefanhaller,

maybe there's some concurrency issue related to this. We already know that there can be cases where spurious errors occur because of this (see for example #3180 (comment), which is slightly different). I don't totally understand what's going on yet, but I can try to look into it more.

They are same. If have pull config, like git config --global pull.rebase true, the warning message becomes to cannot rebase into multiple branches

Meanwhile you probably want to set fetchAll to false, as you said above. Your video shows that fetching your remotes is so slow that you probably don't want to do that all the time.

Exactly! Thanks

Tim-Cao · 2024-01-06T09:54:32Z

Is there any update? @stefanhaller

stefanhaller · 2024-01-06T15:25:40Z

Yes. I just opened a PR (#3202) that fixes the "Cannot rebase onto multiple branches" error. I couldn't reproduce the other error ("You have divergent branches") locally, so I'm not sure if it also fixes that one. Would be great if you could test this, if you can build from source.

Tim-Cao · 2024-01-06T20:05:43Z

Hi @stefanhaller, I tested on the #3202, both Cannot rebase onto multiple branches and You have divergent branches are fixed.

But this issue still happens, do you have another pr to fix it? Additionally, if quit the lazygit during fetching, it will continue when launch lazygit again.

The fetch -all still be tiggered if doesn't have the fetchAll false
Fetching your remotes is so slow that you probably don't want to do that all the time.

stefanhaller · 2024-01-06T22:12:16Z

Hi @stefanhaller, I tested on the #3202, both Cannot rebase onto multiple branches and You have divergent branches are fixed.

That's good to hear, thanks for testing.

But this issue still happens, do you have another pr to fix it? Additionally, if quit the lazygit during fetching, it will continue when launch lazygit again.

The fetch -all still be tiggered if doesn't have the fetchAll false
Fetching your remotes is so slow that you probably don't want to do that all the time.

I don't understand the problem. This is by design, fetch --all is the default, and you can turn it off by setting the fetchAll config to false if you don't want it. What's the problem?

Tim-Cao · 2024-01-07T01:00:03Z

Hi @stefanhaller,
I don't understand why did this change that when run pulling, it will trigger both git pull --no-edit and git fetch -all instead of just git pull --no-edit like before?

stefanhaller · 2024-01-07T09:57:14Z

Hi @stefanhaller, I don't understand why did this change that when run pulling, it will trigger both git pull --no-edit and git fetch -all instead of just git pull --no-edit like before?

That's not what it's doing. When you do a pull, it does not call fetch, only pull.

What you are seeing is the initial background fetch; in the v0.40.2 release the pull command would block until the background fetch is finished, but now on master the pull is allowed to run in parallel to the background fetch. That's why you see both next to each other in the log. You can confirm this by waiting for the fetch to be complete, and then pressing "p"; you'll see that it doesn't call fetch.

Tim-Cao · 2024-01-07T13:03:21Z

Hi @stefanhaller,
Would you mind explaining what does the initial background fetch for?

stefanhaller · 2024-01-07T13:09:32Z

Lazygit has a feature to periodically run git fetch in the background, to keep your remotes up to date without you having to fetch manually. The first of these runs right after startup, from then on it runs every 60 seconds by default. You can change the frequency with the refresher.fetchInterval option, and you can turn it off altogether by setting git.autoFetch to false (it's on by default). See Config.md for more about lazygit's config options.

Tim-Cao · 2024-01-07T13:13:46Z

@stefanhaller
I have 41 remotes, Should I turn it off by default?

stefanhaller · 2024-01-07T13:21:02Z

That's your choice. I thought we agreed above that it makes sense for you to turn off git.fetchAll, since fetching is so slow for you. Whether you also (or instead) want to turn off git.autoFetch is up to you.

Tim-Cao · 2024-01-07T13:26:29Z

@stefanhaller
I tried if disable git.autoFetch on master build, then don't have to disable git.fetchAll, and vice versa. Is that expected?

stefanhaller · 2024-01-07T13:30:57Z

No, fetchAll not only applies to the background fetch, but also to pressing "f" in the files panel to fetch manually, so you still might want to decide for both whether you want them on or off.

Tim-Cao · 2024-01-07T13:34:49Z

@stefanhaller
Do you mean disable git.autoFetch is enough in my case, right?

stefanhaller · 2024-01-07T13:37:39Z

That would solve it for you, yes. Whether that's what you want is not something I can decide for you; you'll have to decide this yourself.

Tim-Cao · 2024-01-07T13:44:33Z

Got it. I really appreciate your help. Look forward to the new release. @stefanhaller

Tim-Cao added the bug Something isn't working label Dec 22, 2023

Tim-Cao changed the title ~~Pull doesn't work on v0.40.2~~ Pull doesn't work from v0.39.1 Dec 27, 2023

Tim-Cao changed the title ~~Pull doesn't work from v0.39.1~~ Pull doesn't work from v0.39.1 since fetchAll is enabled by default Dec 27, 2023

Tim-Cao closed this as completed Jan 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pull doesn't work from v0.39.1 since fetchAll is enabled by default #3181

Pull doesn't work from v0.39.1 since fetchAll is enabled by default #3181

Tim-Cao commented Dec 22, 2023 •

edited

Loading

mark2185 commented Dec 23, 2023

stefanhaller commented Dec 23, 2023

Tim-Cao commented Dec 25, 2023

Tim-Cao commented Dec 25, 2023 •

edited

Loading

stefanhaller commented Dec 26, 2023

Tim-Cao commented Dec 26, 2023

stefanhaller commented Dec 27, 2023

Tim-Cao commented Dec 27, 2023 •

edited

Loading

stefanhaller commented Dec 28, 2023 •

edited

Loading

Tim-Cao commented Dec 28, 2023

stefanhaller commented Dec 29, 2023

Tim-Cao commented Dec 29, 2023 •

edited

Loading

stefanhaller commented Dec 29, 2023

Tim-Cao commented Dec 29, 2023

stefanhaller commented Dec 30, 2023

Tim-Cao commented Dec 31, 2023 •

edited

Loading

stefanhaller commented Dec 31, 2023

Tim-Cao commented Jan 1, 2024 •

edited

Loading

Tim-Cao commented Jan 6, 2024

stefanhaller commented Jan 6, 2024

Tim-Cao commented Jan 6, 2024 •

edited

Loading

stefanhaller commented Jan 6, 2024

Tim-Cao commented Jan 7, 2024 •

edited

Loading

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

Pull doesn't work from v0.39.1 since fetchAll is enabled by default #3181

Pull doesn't work from v0.39.1 since fetchAll is enabled by default #3181

Comments

Tim-Cao commented Dec 22, 2023 • edited Loading

mark2185 commented Dec 23, 2023

stefanhaller commented Dec 23, 2023

Tim-Cao commented Dec 25, 2023

Tim-Cao commented Dec 25, 2023 • edited Loading

stefanhaller commented Dec 26, 2023

Tim-Cao commented Dec 26, 2023

stefanhaller commented Dec 27, 2023

Tim-Cao commented Dec 27, 2023 • edited Loading

stefanhaller commented Dec 28, 2023 • edited Loading

Tim-Cao commented Dec 28, 2023

stefanhaller commented Dec 29, 2023

Tim-Cao commented Dec 29, 2023 • edited Loading

stefanhaller commented Dec 29, 2023

Tim-Cao commented Dec 29, 2023

stefanhaller commented Dec 30, 2023

Tim-Cao commented Dec 31, 2023 • edited Loading

stefanhaller commented Dec 31, 2023

Tim-Cao commented Jan 1, 2024 • edited Loading

Tim-Cao commented Jan 6, 2024

stefanhaller commented Jan 6, 2024

Tim-Cao commented Jan 6, 2024 • edited Loading

stefanhaller commented Jan 6, 2024

Tim-Cao commented Jan 7, 2024 • edited Loading

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

stefanhaller commented Jan 7, 2024

Tim-Cao commented Jan 7, 2024

Tim-Cao commented Dec 22, 2023 •

edited

Loading

Tim-Cao commented Dec 25, 2023 •

edited

Loading

Tim-Cao commented Dec 27, 2023 •

edited

Loading

stefanhaller commented Dec 28, 2023 •

edited

Loading

Tim-Cao commented Dec 29, 2023 •

edited

Loading

Tim-Cao commented Dec 31, 2023 •

edited

Loading

Tim-Cao commented Jan 1, 2024 •

edited

Loading

Tim-Cao commented Jan 6, 2024 •

edited

Loading

Tim-Cao commented Jan 7, 2024 •

edited

Loading