Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow custom cache directories #23

Open
GZGavinZhao opened this issue May 27, 2022 · 1 comment
Open

Allow custom cache directories #23

GZGavinZhao opened this issue May 27, 2022 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@GZGavinZhao
Copy link
Member

Currently solbuild can cache C, C++, and Rust builds. However, in the future, it's likely we'll need to cache more things. For example, the cargo registry (takes 1~2 minutes to pull the index), Go build cache, etc.

Therefore, I propose the ability for packagers to specify additional cache entries using /etc/solbuild/<profile>.conf. The structure can look like this:

# By default, sccache and ccache are provided.
[cache]
caches = ["ccache", "sccache", "cargo"]

[cache.cargo]
# Relative to the $BUILDUSER's $HOME directory, which should be enough for most conditions.
# This can be further discussed.
internalDir = ".cargo"
# For legacy builds. Require a full path.
legacyInternalDir = "/root/.cargo"

The benefits of this approach are that it does not affect existing cache (e.g. for the build server that doesn't want any caching), and it is very simple (we don't need to patch YPKG).

To achieve this, the cache directories' layout needs to be changed. Currently, we have /var/lib/solbuild/ccache and /var/lib/solbuild/sccache. For easier cache management, all build cache will be stored in /var/lib/solbuild/cache/<cache-name>. For example, the ccache directory will go to /var/lib/solbuild/ccache, and the cargo registry directory (in the example above) will go to /var/lib/solbuild/cargo. When deleting cache (solbuild dc), we can simply delete /var/lib/solbuild/cache.

If this is not possible, I suggest at least refactor the code so that in the future, we can add additional caches as easy as constructing a new struct (Cache) and add it to a global variable called caches, instead of writing 10 new methods by hand. I already implemented this locally and have been using it for a week, so I can verify this will not break anything. Unfortunately, I recently wiped my PC without pushing my commits to the remote, so I don't have them ready right now, but I am happy to rewrite them since they're still refresh in my head.

@joebonrichie
Copy link
Contributor

We'll definitely want to cache more stuff in the future (the cargo cache comes to mind as you mentioned). This proposal makes general sense to me to avoid mindless copy pasta code of directories into the overlay.

For example, the ccache directory will go to /var/lib/solbuild/ccache, and the cargo registry directory (in the example above) will go to /var/lib/solbuild/cargo. When deleting cache (solbuild dc), we can simply delete /var/lib/solbuild/cache.

Assuming this is a typo e.g. /var/lib/solbuild/cache/cargo /var/lib/solbuild/cache/ccache

For compatibility we'll need to make sure that solbuild dc will also remove the cache of old directory locations if they exist (unless you can think of a better solution).

W.R.T the cargo cache specifically i'm not sure how to handle that. Personally i do not have the cargo cache in my home directory so there is nothing to cache from. It may make more sense to cache it from solbuild itself when building a package. Although, i am not sure if updating a very old cargo cache index is quicker than downloading the initial index from scratch. We may also need to think about caching similar network based tools such as npm, yarn, java build tools as well.

Feel free to send a PR for the cache refactoring for now, as that will cleanup a lot of copypasta, and we can work out details to cache networking based tools in the future.

@ermo ermo added the enhancement New feature or request label Feb 27, 2024
@ermo ermo added this to the 1.6.x milestone Feb 27, 2024
@silkeh silkeh modified the milestones: 1.6.x, 1.7.0 Feb 28, 2024
@silkeh silkeh modified the milestones: 1.7.0, 1.8.0 Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

No branches or pull requests

4 participants