Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Source directory option for tests #191

Open
wants to merge 6 commits into
base: devel
Choose a base branch
from

Conversation

hagertnl
Copy link
Contributor

Added the ability to have a Source directory under each test.

This directory is copied second and will override any files in the main Source directory.

This is currently implemented with a conditional to allow for Python < 3.8 to still run, with the intention to remove that conditional once all OLCF machines use Python >3.8 in the coming years.

@hagertnl
Copy link
Contributor Author

Closes #190

@hagertnl hagertnl linked an issue Dec 19, 2024 that may be closed by this pull request
@hagertnl hagertnl requested a review from ddietz89 December 19, 2024 19:06
@hagertnl
Copy link
Contributor Author

Documentation has been updated to reflect the new directory, and I have added a new environment variable to allow referencing to the source directory.

@hagertnl
Copy link
Contributor Author

I can confirm that sym-links also work!! This worked when I tried it:

$ ls -lh
total 12K
drwxr-xr-x 7 hagertnl hagertnl 4.0K Dec 19 17:02 Run_Archive
drwxr-xr-x 2 hagertnl hagertnl 4.0K Dec 19 17:02 Scripts
lrwxrwxrwx 1 hagertnl hagertnl   31 Dec 19 17:01 Source -> ../test_test_source_dir/Source/
drwxr-xr-x 7 hagertnl hagertnl 4.0K Dec 19 17:02 Status

This makes it much easier to maintain differences if needed by more than 1 test.

@hagertnl
Copy link
Contributor Author

I'm finding some sort of problem with the old-Python implementation of this:

cp: cannot overwrite directory '/lustre/orion/proj-shared/stf243/hagertnl/harness_sspace/borg/12.27.24-09.45/amdgpu_checkout/01_allocate_full_hbm_cray/1735310728.4993293/build_directory' with non-directory
Encountered an error copying a test's Source directory from /lustre/orion/stf243/proj-shared/hagertnl/Scratch/harness-dev/frontier/amdgpu_checkout/01_allocate_full_hbm_cray/Source to /lustre/orion/proj-shared/stf243/hagertnl/harness_sspace/borg/12.27.24-09.45/amdgpu_checkout/01_allocate_full_hbm_cray/1735310728.4993293/build_directory

If I load a newer Python version, then it works as expected. Working to reproduce now.

@hagertnl
Copy link
Contributor Author

The sym-links that I previously reported as working appear to no longer be working 🤔 . It's possible that I only checked the sym-links for the newer Python version and that I didn't actually verify it for the shell implementation.

@hagertnl
Copy link
Contributor Author

Okay, so the problem was that I had sym-links going to & from multiple test source directories. For example, maintain a Makefile in one test, then sym-link to it from all other tests of a certain type. We need that sym-link to be dereferenced in that case, so I have set the following behavior:

sym-links in <app>/Source are preserved when copied
sym-links in <app>/<test>/Source are dereferenced

The <app>/Source directory should be stand-alone (not referencing any other paths outside of Source), while it is feasible that a group of test's may want to share the same Makefile, job template or check script.

I've pushed these changes to the default behavior now, and confirmed that it is working for my newly-refactored amdgpu_checkout tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: test-specific source directory
1 participant