Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(fs/git): only fetch known or newly requested references #3100

Merged
merged 5 commits into from
May 21, 2024

Conversation

GeorgeMac
Copy link
Member

@GeorgeMac GeorgeMac commented May 21, 2024

This is an optimization to ensure that we only ever fetch either tracked or newly requested references from the upstream.
Prior to this change, we always fetched all references from the upstream.
Now instead, we always pull the main reference, plus any newly requested refs or previously fetched refs still in the LRU.

I just added an additional change to this, to ensure that when a branch is supplied (only a branch, not a tag or SHA), that we also perform a single branch clone. So that we don't start off with a full clone.

One user reported to us that a new branch in their upstream was unnecessarily fetched and this caused memory pressure on their instance and it OOMed.
Their relevant feedback on this change:

I tracked down the culprit of our sudden increase in resource usage to a random in-development branch of our repo that had just been pushed to the remote.
Since flipt requires the specific git ref for determining which branch / tag to consume, might it be possible to perform a single-ref clone of the repo, rather than cloning all branches?

@GeorgeMac GeorgeMac requested a review from a team as a code owner May 21, 2024 09:42
@GeorgeMac GeorgeMac force-pushed the gm/optimize-git-fetch branch from 03e0069 to b36e3f1 Compare May 21, 2024 09:45
Copy link
Collaborator

@markphelps markphelps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm !

@GeorgeMac GeorgeMac force-pushed the gm/optimize-git-fetch branch from 4be45ca to e1cff0f Compare May 21, 2024 12:59
@GeorgeMac GeorgeMac enabled auto-merge May 21, 2024 13:01
@GeorgeMac GeorgeMac merged commit 82fa7a2 into main May 21, 2024
29 checks passed
@GeorgeMac GeorgeMac deleted the gm/optimize-git-fetch branch May 21, 2024 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants