Testing/manager #14
Open
Testing/manager #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (LLNL#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (LLNL#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (LLNL#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (LLNL#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (LLNL#436) * Release/1.10.2 (LLNL#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (LLNL#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (LLNL#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (LLNL#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (LLNL#444) * bugfix/sphinx-5.3.0-requirement (LLNL#446) * Version/1.10.3 (LLNL#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (LLNL#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (LLNL#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (LLNL#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (LLNL#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (LLNL#436) * Release/1.10.2 (LLNL#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (LLNL#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (LLNL#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (LLNL#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (LLNL#444) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (LLNL#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <[email protected]> * release/1.11.0 (LLNL#448) * bugfix/skewed-sample-hierarchy (LLNL#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (LLNL#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (LLNL#454) * Add Pytest Fixtures to Test Suite (LLNL#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (LLNL#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (LLNL#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (LLNL#459) * docs/conversion-to-mkdocs (LLNL#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (LLNL#463) * feature/revamped status (LLNL#464) * feature/new-status (LLNL#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (LLNL#451) * Version/1.11.0 (LLNL#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (LLNL#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (LLNL#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (LLNL#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (LLNL#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (LLNL#436) * Release/1.10.2 (LLNL#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (LLNL#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (LLNL#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (LLNL#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (LLNL#444) * bugfix/sphinx-5.3.0-requirement (LLNL#446) * Version/1.10.3 (LLNL#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (LLNL#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (LLNL#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (LLNL#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (LLNL#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (LLNL#436) * Release/1.10.2 (LLNL#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (LLNL#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (LLNL#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (LLNL#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (LLNL#444) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (LLNL#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <[email protected]> * release/1.11.0 (LLNL#448) * bugfix/skewed-sample-hierarchy (LLNL#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <[email protected]> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (LLNL#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (LLNL#454) * Add Pytest Fixtures to Test Suite (LLNL#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (LLNL#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (LLNL#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (LLNL#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <[email protected]> Co-authored-by: Joe Koning <[email protected]> * feature/queue info (LLNL#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (LLNL#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <[email protected]> Co-authored-by: Joe Koning <[email protected]> * release/1.12.0 (LLNL#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (LLNL#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (LLNL#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (LLNL#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (LLNL#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (LLNL#475) * bugfix/deep-merge-existing-keys (LLNL#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (LLNL#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (LLNL#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (LLNL#479) * bugfix/status_nested_workspace (LLNL#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (LLNL#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (LLNL#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (LLNL#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (LLNL#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (LLNL#486) * Modifying task id to include directory --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <[email protected]> Co-authored-by: Joe Koning <[email protected]> Co-authored-by: Jane Herriman <[email protected]> Co-authored-by: Luc Peterson <[email protected]>
…dis to store env dict
…f config not being created yet
* remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue
* update version to 1.12.2 * fix style issue
* remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <[email protected]> Co-authored-by: Ryan Lee <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.