Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update for TestTokenQueries.py and TestWebserver.py #975

Merged
merged 12 commits into from
Sep 10, 2024
Merged

Update for TestTokenQueries.py and TestWebserver.py #975

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
d0bc264
Added dynamic conda environment name to assertion log messages
Aug 21, 2024
1dc4275
Changed TestWebserver.py to work with both config files and ENV varia…
Aug 22, 2024
7ac46aa
Logging CONDA_DEFAULT_ENV to investigate failed tests
Aug 22, 2024
32681cf
Corrected 1st log message for local emission environment
Aug 22, 2024
769346c
Correcting import for restoreOriginalEnvVars
Aug 23, 2024
39885a3
Added fourth message for running tests in local docker container
Aug 23, 2024
9ab6358
Revert "Changed TestWebserver.py to work with both config files and E…
Aug 23, 2024
1a13a49
Added WEBSERVER_NOT_FOUND_REDIRECT to config map
Aug 23, 2024
0efbdc8
Simplified tests with two log messages for four test environments
Aug 26, 2024
9bba6c8
Logging retrieved db config values before using default values
Aug 29, 2024
b2aefce
✅ Unify the expected messages through redaction
shankari Sep 1, 2024
6368598
Added missing import for 'os' module
Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion emission/core/get_database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
config = ecbc.get_config('conf/storage/db.conf',
{"DB_HOST": "timeseries.url", "DB_RESULT_LIMIT": "timeseries.result_limit"})

print("Retrieved config %s" % config)
url = config.get("DB_HOST", "localhost")
result_limit = config.get("DB_RESULT_LIMIT", 250000)
print("Retrieved config: {'DB_HOST': '%s', 'DB_RESULT_LIMIT': '%s'}" % (url, result_limit))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MukuFlash03 the problem with printing these values after the config.get calls is that it is not sufficient for debugging. You can see the final value of the variables here, but you also see them on line 34. The point behind putting the "Retrieved config" before the config.get is so that we could see the inputs from the environment before they were modified by the defaults.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging these values before using default values.
Notes in commit message.


try:
parsed=pymongo.uri_parser.parse_uri(url)
Expand Down
4 changes: 3 additions & 1 deletion emission/integrationTests/storageTests/TestMongodbAuth.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
import importlib
import os

import emission.tests.common as etc

class TestMongodbAuth(unittest.TestCase):
def setUp(self):
self.admin_default = pymongo.MongoClient('localhost', username="admin",
Expand All @@ -53,7 +55,7 @@ def setUp(self):
def tearDown(self):
self.admin_auth.command({"dropAllUsersFromDatabase": 1})
logging.debug("Deleting test db environment variables")
ecc.restoreOriginalEnvVars(self.originalDBEnvVars, self.modifiedEnvVars)
etc.restoreOriginalEnvVars(self.originalDBEnvVars, self.modifiedEnvVars)
logging.debug("Finished restoring original db environment variables")
logging.debug("Restored original values are = %s" % self.originalDBEnvVars)
try:
Expand Down
3 changes: 2 additions & 1 deletion emission/net/api/cfc_webapp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,8 @@

config = ecbc.get_config('conf/net/api/webserver.conf',
{"WEBSERVER_PORT": "server.port", "WEBSERVER_TIMEOUT": "server.timeout",
"WEBSERVER_AUTH": "server.auth", "WEBSERVER_AGGREGATE_CALL_AUTH": "server.aggregate_call_auth"})
"WEBSERVER_AUTH": "server.auth", "WEBSERVER_AGGREGATE_CALL_AUTH": "server.aggregate_call_auth",
"WEBSERVER_NOT_FOUND_REDIRECT": "paths.404_redirect"})
server_port = config.get("WEBSERVER_PORT", 8080)
socket_timeout = config.get("WEBSERVER_TIMEOUT", 3600)
auth_method = config.get("WEBSERVER_AUTH", "skip")
Expand Down
20 changes: 8 additions & 12 deletions emission/tests/storageTests/TestTokenQueries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,24 +168,20 @@ def test_run_script_show(self):
esdt.insert({'token':'y'})
esdt.insert({'token':'z'})
sp = subprocess.run(["python3", "bin/auth/insert_tokens.py", "--show"], capture_output=True)
# The first message is displayed when we run tests locally
# The second is displayed when we run in the CI/CD, but with the local install
# The third is displayed when we run in the docker CI since the `DB_HOST` is set to `db`
# The first message is displayed when we run tests locally or run in the CI/CD, but with the local install
# The second is displayed when we run tests locally in a docker container or run in the docker CI; the `DB_HOST` is set to `db` by default
self.assertIn(sp.stdout,
[b'Retrieved config {\'DB_HOST\': \'localhost\', \'DB_RESULT_LIMIT\': 250000}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL localhost\nx\ny\nz\n',
b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config {\'PYTHONPATH\': \'.\', \'PATH\': \'/home/runner/miniconda-23.5.2/envs/emissiontest/bin:/home/runner/miniconda-23.5.2/condabin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/runner/.dotnet/tools\', \'LC_CTYPE\': \'C.UTF-8\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL localhost\nx\ny\nz\n',
b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config {\'PYTHONPATH\': \'.\', \'DB_HOST\': \'db\', \'PATH\': \'/root/miniconda-23.5.2/envs/emissiontest/bin:/root/miniconda-23.5.2/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\', \'LC_CTYPE\': \'C.UTF-8\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL db\nx\ny\nz\n'
[b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config: {\'DB_HOST\': \'localhost\', \'DB_RESULT_LIMIT\': \'250000\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL localhost\nx\ny\nz\n',
b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config: {\'DB_HOST\': \'db\', \'DB_RESULT_LIMIT\': \'250000\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL db\nx\ny\nz\n'
])

def test_run_script_empty(self):
sp = subprocess.run(["python3", "bin/auth/insert_tokens.py"], capture_output=True)
# The first message is displayed when we run tests locally
# The second is displayed when we run in the CI/CD, but with the local install
# The third is displayed when we run in the docker CI since the `DB_HOST` is set to `db`
# The first message is displayed when we run tests locally or run in the CI/CD, but with the local install
# The second is displayed when we run tests locally in a docker container or run in the docker CI; the `DB_HOST` is set to `db` by default
self.assertIn(sp.stdout,
[b'Retrieved config {\'DB_HOST\': \'localhost\', \'DB_RESULT_LIMIT\': 250000}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL localhost\nPlease provide the script with an argument. Use the "--help" option for more details\n',
b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config {\'PYTHONPATH\': \'.\', \'PATH\': \'/home/runner/miniconda-23.5.2/envs/emissiontest/bin:/home/runner/miniconda-23.5.2/condabin:/snap/bin:/home/runner/.local/bin:/opt/pipx_bin:/home/runner/.cargo/bin:/home/runner/.config/composer/vendor/bin:/usr/local/.ghcup/bin:/home/runner/.dotnet/tools:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/runner/.dotnet/tools\', \'LC_CTYPE\': \'C.UTF-8\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL localhost\nPlease provide the script with an argument. Use the "--help" option for more details\n',
b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config {\'PYTHONPATH\': \'.\', \'DB_HOST\': \'db\', \'PATH\': \'/root/miniconda-23.5.2/envs/emissiontest/bin:/root/miniconda-23.5.2/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\', \'LC_CTYPE\': \'C.UTF-8\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL db\nPlease provide the script with an argument. Use the "--help" option for more details\n'
[b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config: {\'DB_HOST\': \'localhost\', \'DB_RESULT_LIMIT\': \'250000\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL localhost\nPlease provide the script with an argument. Use the "--help" option for more details\n',
b'Config file not found, returning a copy of the environment variables instead...\nRetrieved config: {\'DB_HOST\': \'db\', \'DB_RESULT_LIMIT\': \'250000\'}\nURL not formatted, defaulting to "Stage_database"\nConnecting to database URL db\nPlease provide the script with an argument. Use the "--help" option for more details\n'
])

#test that no two options can be used together
Expand Down
Loading