Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix sky key secret creation so it won't break release testing locally #1594

Merged
merged 1 commit into from
Dec 15, 2024

Conversation

BelSasha
Copy link
Contributor

@BelSasha BelSasha commented Dec 15, 2024

The problem:
ssh-sky-key is not being deleted during den clean up (which makes sense, it should not have). Therefore, there might be a mismatch between the ssh-sky-key values saved as a den secret, and the local ssh-sky-key values. (For example the ssh-sky-key values saved in den might belong to user A who created and saved this secret for den_tester and the local values might belong to user B who is running release testing with den_tester). This is the explanation to the following issue I was seeing:

ERROR tests/test_resources/test_modules/test_module.py::TestModule::test_module_from_factory[static_cpu_pwd_cluster_den_launcher] - ConnectionError: Could not reach testing-b83cbcb4-14fb-4c7b-ae13-7c73b62b8d2c-20241211-173742-den-aws-cpu-password 44.204.57.6. Is cluster up?
(base) sashabelousovrh@Alexandras-MacBook-Pro runhouse % runhouse server status /den_tester/testing-e91feebf-072b-4350-bb67-f75750c1fdcf-20241211-175620-den-aws-cpu-password

INFO | 2024-12-11 17:58:46 | runhouse.resources.hardware.cluster:1832 | Running command on testing-e91feebf-072b-4350-bb67-f75750c1fdcf-20241211-175620-den-aws-cpu-password: echo "hello"
INFO | 2024-12-11 17:58:47 | runhouse.resources.hardware.cluster:1832 | Running command on testing-e91feebf-072b-4350-bb67-f75750c1fdcf-20241211-175620-den-aws-cpu-password: echo "hello"
Cluster /den_tester/testing-e91feebf-072b-4350-bb67-f75750c1fdcf-20241211-175620-den-aws-cpu-password is not up. If it's an on-demand cluster, you can run `runhouse cluster up 
/den_tester/testing-e91feebf-072b-4350-bb67-f75750c1fdcf-20241211-175620-den-aws-cpu-password` to bring it up automatically.

The fix probably should be creating a new ssh-sky-key each time we are creating a cluster using the test cluster fixtures. This way we could parallelly run tests using den_tester (CI, few runhouse members running release tests locally etc), without getting the above error, since now the values of the local sky key and the one saved in den will match.

@BelSasha BelSasha changed the title fix sky key secret creation so it won't break release testing locally [WIP] fix sky key secret creation so it won't break release testing locally Dec 15, 2024
@BelSasha BelSasha force-pushed the sb/fix-sky-key-issue-for-tests-den-launcher branch from 9599b77 to 7f6835a Compare December 15, 2024 13:44
@BelSasha BelSasha force-pushed the sb/fix-sky-key-issue-for-tests-den-launcher branch from 7f6835a to 5674950 Compare December 15, 2024 14:21
@@ -107,7 +107,15 @@ def supported_providers():

@classmethod
def sky_secret(cls):
from tests.fixtures.resource_fixtures import create_folder_path

secrets_name = "ssh-sky-key"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of changing logic to accomodate the test, we can save "ssh-sky-key" as a constant, and then just overwrite it in the test

rh.constants.SSH_SKY_KEY = ""

@BelSasha BelSasha force-pushed the sb/fix-sky-key-issue-for-tests-den-launcher branch 2 times, most recently from e581892 to 87832ad Compare December 15, 2024 14:49
@BelSasha BelSasha force-pushed the sb/fix-sky-key-issue-for-tests-den-launcher branch from 87832ad to 79832f6 Compare December 15, 2024 14:54
@BelSasha BelSasha requested a review from jlewitt1 December 15, 2024 14:56
@BelSasha BelSasha marked this pull request as ready for review December 15, 2024 14:56
@BelSasha BelSasha changed the title [WIP] fix sky key secret creation so it won't break release testing locally fix sky key secret creation so it won't break release testing locally Dec 15, 2024
@BelSasha BelSasha merged commit ca359a6 into main Dec 15, 2024
10 of 11 checks passed
@BelSasha BelSasha deleted the sb/fix-sky-key-issue-for-tests-den-launcher branch December 15, 2024 15:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants