Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Google Data Access Tests #106

Merged
merged 18 commits into from
Sep 24, 2024
Merged

Migrate Google Data Access Tests #106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
db4c759
Migrate Google Data Access Tests
atharvar28 Aug 30, 2024
7b7525f
Merge branch 'master' into chore/migrate_google_data_tests
krishnaa05 Sep 3, 2024
ecd5ece
Merge branch 'master' into chore/migrate_google_data_tests
krishnaa05 Sep 6, 2024
1a41c3c
Merge branch 'master' into chore/migrate_google_data_tests
krishnaa05 Sep 9, 2024
7e0f11f
minor fixes
krishnaa05 Sep 13, 2024
87f45f1
Update test_google_data_access.py
krishnaa05 Sep 13, 2024
c05ad65
fixes
krishnaa05 Sep 13, 2024
653c096
code cleanup
krishnaa05 Sep 16, 2024
d851069
Google Storage Access
krishnaa05 Sep 16, 2024
506e64b
Merge branch 'master' into chore/migrate_google_data_tests
krishnaa05 Sep 16, 2024
d59cd33
changes for google buckets creation
krishnaa05 Sep 23, 2024
53bf290
Fix Google Link Failures
krishnaa05 Sep 23, 2024
feca7f2
Merge branch 'master' into chore/migrate_google_data_tests
krishnaa05 Sep 23, 2024
04f8b0e
Merge branch 'master' into chore/migrate_google_data_tests
krishnaa05 Sep 23, 2024
d9f0ea6
Update login.py
krishnaa05 Sep 23, 2024
58c3e57
change for OIDC client
krishnaa05 Sep 24, 2024
76d78fe
minor fixes
krishnaa05 Sep 24, 2024
cbc5a2a
minor fix
krishnaa05 Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ gen3-integration-tests/test_data/lock
gen3-integration-tests/test_data/configuration
gen3-integration-tests/test_data/graph_data
gen3-integration-tests/test_data/fence_client
gen3-integration-tests/test_data/google_creds
157 changes: 18 additions & 139 deletions .secrets.baseline
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,143 +112,6 @@
}
],
"results": {
"gen3-integration-tests/test_data/fence_clients/client_rotate_creds.txt": [
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/client_rotate_creds.txt",
"hashed_secret": "53709eebcf139fb2ca486c04f065070e321c102a",
"is_verified": false,
"line_number": 1
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/client_rotate_creds.txt",
"hashed_secret": "fd145b883f4f94e6baad59399cc542f7ac49a5d8",
"is_verified": false,
"line_number": 1
}
],
"gen3-integration-tests/test_data/fence_clients/clients_creds.txt": [
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "17b898c11407d972798982c364c70dd03bd0fb62",
"is_verified": false,
"line_number": 1
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "ab3e0294932fe7abad9fb58c3e41758b90e70595",
"is_verified": false,
"line_number": 1
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "ff0f231c5344e69d56e1e554b623949911279ca6",
"is_verified": false,
"line_number": 2
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "01662b9143b84b0eaae23336237cde7bf39ab042",
"is_verified": false,
"line_number": 3
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "f1cce78768a849beac11d93ec43ba565e819c0da",
"is_verified": false,
"line_number": 3
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "95c46bac73679ff696a2843f65df7964cdce78ab",
"is_verified": false,
"line_number": 4
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "a73f004e3889b1342751d4670b09dc15cfc51327",
"is_verified": false,
"line_number": 4
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "6014178b480b477133815cca5276dc03361d9310",
"is_verified": false,
"line_number": 5
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "7c9aa25179da69db33502af670476fc20182093c",
"is_verified": false,
"line_number": 5
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "c4cd1dd1f8cd6186c0ed46bc29a784c7ec26b2c9",
"is_verified": false,
"line_number": 6
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "d8edfaea81724f5a10855691d92f61de66280269",
"is_verified": false,
"line_number": 6
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "9ed12c9978fcc9cebc65cd1fc5ef366de62c9734",
"is_verified": false,
"line_number": 7
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "e37ec01ad0109eec802dfbd617e162080c835316",
"is_verified": false,
"line_number": 7
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "3c4d365b05bba7dca96ef316a7596fa01b108abf",
"is_verified": false,
"line_number": 8
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "9212e6bd53da0a2bcb0a41119ee2378d9b0d3e2c",
"is_verified": false,
"line_number": 8
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "089f848c95240e6ec6b14a4da935253a00555a99",
"is_verified": false,
"line_number": 9
},
{
"type": "Base64 High Entropy String",
"filename": "gen3-integration-tests/test_data/fence_clients/clients_creds.txt",
"hashed_secret": "57be9b25b765c9c7b16c05481e1d05885eff83a8",
"is_verified": false,
"line_number": 9
}
],
"gen3-integration-tests/test_data/study_registration/study.json": [
{
"type": "Hex High Entropy String",
Expand Down Expand Up @@ -435,6 +298,22 @@
"is_secret": false
}
],
"gen3-integration-tests/tests/test_google_data_access.py": [
{
"type": "Hex High Entropy String",
"filename": "gen3-integration-tests/tests/test_google_data_access.py",
"hashed_secret": "4394745783e204312dff5d8d3594231ea79deca2",
"is_verified": false,
"line_number": 20
},
{
"type": "Hex High Entropy String",
"filename": "gen3-integration-tests/tests/test_google_data_access.py",
"hashed_secret": "af7cfa6ae665ad7a29daa914aa2326f4e36480f7",
"is_verified": false,
"line_number": 27
}
],
"gen3-integration-tests/tests/test_indexing_page.py": [
{
"type": "Hex High Entropy String",
Expand All @@ -451,7 +330,7 @@
"filename": "gen3-integration-tests/tests/test_oidc_client.py",
"hashed_secret": "62bd0c4d3a6b445b13212d23500a7f0916757c3e",
"is_verified": false,
"line_number": 145
"line_number": 147
}
],
"gen3-integration-tests/tests/test_presigned_url.py": [
Expand Down Expand Up @@ -484,5 +363,5 @@
}
]
},
"generated_at": "2024-09-06T16:07:37Z"
"generated_at": "2024-09-24T17:48:54Z"
}
15 changes: 13 additions & 2 deletions gen3-integration-tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

load_dotenv()
requires_fence_client_marker_present = False
requires_google_bucket_marker_present = False


class XDistCustomPlugin:
Expand Down Expand Up @@ -49,17 +50,27 @@ def _split_scope(self, nodeid):

def pytest_collection_finish(session):
global requires_fence_client_marker_present
global requires_google_bucket_marker_present
# Iterate through the collected test items
if not hasattr(session.config, "workerinput"):
for item in session.items:
# Access the markers for each test item
markers = item.keywords
for marker_name, marker in markers.items():
if marker_name == "requires_fence_client":
if (
marker_name == "requires_fence_client"
and requires_fence_client_marker_present == False
):
setup.get_fence_client_info()
setup.get_fence_rotated_client_info()
requires_fence_client_marker_present = True
return
if (
marker_name == "requires_google_bucket"
and requires_google_bucket_marker_present == False
):
# Create and Link Google Test Buckets
setup.setup_google_buckets()
requires_google_bucket_marker_present = True


@pytest.fixture(scope="session", autouse=True)
Expand Down
14 changes: 9 additions & 5 deletions gen3-integration-tests/gen3_ci/jenkins_jobs/ci-only-create-link-google-test-buckets.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ spec:
}
stage('Create Link Google Test Buckets') {
steps {
dir("create-link-google-test-buckets"){
dir("ci-only-create-link-google-test-buckets"){
script {
sh '''#!/bin/bash +x
set -e
Expand All @@ -110,12 +110,16 @@ spec:
source $GEN3_HOME/gen3/gen3setup.sh

# Create Google Test Buckets
kubectl exec $(gen3 pod fence \${NAMESPACE}) -- fence-create google-bucket-create --unique-name dcf-integration-qa --google-project-id dcf-integration --project-auth-id QA --public False
kubectl exec $(gen3 pod fence \${NAMESPACE}) -- fence-create google-bucket-create --unique-name dcf-integration-test --google-project-id dcf-integration --project-auth-id test --public False
kubectl exec -n ${KUBECTL_NAMESPACE} $(gen3 pod fence \${NAMESPACE}) -- fence-create google-bucket-create --unique-name dcf-integration-qa --google-project-id dcf-integration --project-auth-id QA --public False
kubectl exec -n ${KUBECTL_NAMESPACE} $(gen3 pod fence \${NAMESPACE}) -- fence-create google-bucket-create --unique-name dcf-integration-test --google-project-id dcf-integration --project-auth-id test --public False

# Link phs ids to existing buckets
kubectl exec $(gen3 pod fence \${NAMESPACE}) -- fence-create link-bucket-to-project --project_auth_id phs000179 --bucket_id dcf-integration-qa --bucket_provider google
kubectl exec $(gen3 pod fence \${NAMESPACE}) -- fence-create link-bucket-to-project --project_auth_id phs000178 --bucket_id dcf-integration-test --bucket_provider google
kubectl exec -n ${KUBECTL_NAMESPACE} $(gen3 pod fence \${NAMESPACE}) -- fence-create link-bucket-to-project --project_auth_id phs000179 --bucket_id dcf-integration-qa --bucket_provider google
kubectl exec -n ${KUBECTL_NAMESPACE} $(gen3 pod fence \${NAMESPACE}) -- fence-create link-bucket-to-project --project_auth_id phs000178 --bucket_id dcf-integration-test --bucket_provider google

# Run usersync
gen3 job run usersync ADD_DBGAP true
kubectl -n ${KUBECTL_NAMESPACE} wait --for=condition=complete --timeout=-1s jobs/usersync
'''
}
}
Expand Down
Loading
Loading