Add support for caching multiple workflows #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When the project.toml does not refer to package dependencies, the 'name" may be missing.
For the use case of distributed systems with shared depot path
Distributed systems may benefit from having a depot path with already precompiled cache objects
To avoid copying populated project folders, especially when the .toml file is referring to a project and not a package dependency
Multiple users may benefit from sharing the depot path. They can now populate the same folder by providing their workflows into the build function via multiple dispatch.
Added docstrings for both versions of build(), and some other explaining comments
Added example with precompilation and custom depot path
Added few comments to explain second readme example
joshday
requested changes
Jul 26, 2024
To copy full packages into dev/
Added message to explain that build() copies everything that coexists with the Project.toml file. This is to avoid copying large files in the case of user-made projects/applications (not packages).
Added the argument force=true inside cp, because the first test set is not compliant anymore with the previous version, throwing an error `ArgumentError: "directory" exists. `force=true` is required to remove "directory" before copying.
Added three project toml files that consist of the packages URI, Unzip and BitFlags, to test capabilities of the DepotDelivery.build function.
Added two test sets to test that the packages were installed correctly and also that the cache files exist in the host machine. The test consists on installing and precompiling three packages: URIs, Unzip and BitFlags that exist in different Project.toml files inside the directory MultipleWorkflows/
Updated number of required precompiled files to zero
|
@joshday , I have implemented some changes and added a test for my use case. All tests seem to pass, let me know if anything else is needed. |
|
Looks great! Thanks for the contribution. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request addresses a need that we have at CERN (high energy physics) where we need to prepare Julia artifacts to support multiple workflows. The artifacts are prepared and then installed on CernVM-FS - a high performance distributed read-only file system used by the LHC experiments (and many other scientific communities).
The changes made implement the following features:
DEPOT_PATHdestination.Project.tomlfiles that are generated by the project dependencies of an application (which may not be a package). This support includes being able to create multiple subdirectories of the target depot path and copying only the relevant (.toml) files to the final destination.To make use of this package at CERN, we gather a list of directories that contain
Project.tomlfiles for a number of relevant workflows. Using DepotDelivery, we build all the dependencies into a single depot path inside CernVM-FS, with the new precompiled flag set to true. After publishing this depot path to CernVM-FS these files are visible to machines connected to the grid. This greatly reduces the startup time of applications running in our distributed computing infrastructure.See the updated readme file for an example (current usage of the script is completely unaffected)
Thanks for creating such a useful script. With these extensions we can really nicely satisfy our use case.