Release 1.4.0 #416
Merged
Release 1.4.0 #416
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r formatting and labeling PR's (#136) * updated to include new conventions for formatting and labeling PR's * added checklist item to correct base branch --------- Co-authored-by: Linda Lovero-Waterhouse <[email protected]>
Typo Fix in Security Log Dashboard | Sercurity to Security
* Add a markdown file for dashboard descriptions, link new file in README.md (#138) * Add a markdown file for dashboard descriptions ## 🗣 Description ## <!-- Describe the "what" of your changes in detail. --> Add dashboard-descriptions.md in /docs/markdown/reference. Add a link to this file within the main README.md's table of contents. ### 💭 Motivation and context <!-- Why is this change required? --> <!-- What problem does this change solve? How did you solve it? --> <!-- Mention any related issue(s) here using appropriate keywords such --> <!-- as "closes" or "resolves" to auto-close them on merge. --> The LME repository does not have a location for dashboard descriptions. ## 🧪 Testing <!-- How did you test your changes? How could someone else test this PR? --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> N/A ## ✅ Pre-approval checklist ## - [x] Changes are limited to a single goal **AND** the title reflects this in a clear human readable format - [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document. - [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR) - [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide). - [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR. ## ✅ Post-merge Checklist - [x] Squash all commits into one PR level commit - [x] Delete the branch to keep down number of branches * Update README.md to include dashboard-descriptions.md * Update wording for computer software overview dashboard * Fix some grammatical changes in dashboard-descriptions.md * Release 1.3.1 merge into main (#154) * Update retention function to fix retention policy bug (#143) * Updated troubleshooting guide to account for index management (#134) * Update upgrading.md to account for 1.3.1 (#151) * Update upgrading.md * Update upgrading.md --------- Co-authored-by: Andrew Arz <[email protected]> * Fixes dashboard update not importing on fresh install (#167) (#169) * Fixes dashboard update not importing on fresh install #165 * Update upgrading.md to include status on v1.3.2, along with revisions to the document overall * remove step 4 from upgrading.md; add additional instructions for v1.3.2 --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> * Add proof of concept selenium tests * Correct the script name in the doc string * User Security Selenium Tests for No Results Panels * First full selenium test. Currently just User Security * WIP User HR * Completed all dashboards. Requires testing now * Debugging a couple unit tests that error out. Two left * Cut dev comments Co-authored-by: Alden Hilton <[email protected]> * Switched script to headless mode * added switch for headless, detached, and debug mode. Bug where driver.quit does not close window. * Install LME in the testbed from a single script (#150) * Adding the configure scripts * Add scripts to zip and copy to a container for downloading in the server * Grab the expiry time properly in copy file * Overwrite the blob if it exists * Add the script to download file into DC * Script that unzips the files in a container * Adds username argument to download files * Add script to run scripts in container * Adds username argument to gpo script * Modifies the url name in the client GPO * Adds the functionality for chapter 1 and first half of chapter 2 * Imports the sysmon GPO * Update the variables for sysmon gpo * Name the scripts so they are grouped together in a listing * Echos the file download url * Expands the domain name correctly in create ou * Write the url output of copy file to container to a different output stream * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Echos the file download url * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all of the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Adds InstallTestbed instructions to Readme.md * Modifies parameters to be pascal case * ls1 not being set on DC1 * Adds Linux Only install to SetupTestbed * Remove separate linux only script * Update testing/Readme.md Co-authored-by: Alden Hilton <[email protected]> * Make number of clients consisten between scripts * Add ports for elk stack for testing * Update readmes to change ResourceGroupName to ResourceGroup * Adds a switch to install linux only * Adds simple tests to check install * Removes the error if the old configure zip is not found. * Adds variables to linux tests run command * Move credential extraction to lib for use by other scripts. * Adds npm for other testing * Adds latest version of nodejs for testing * Make output.log readable for tests * Add the -m parameter in the testing readme * Download the latest version or a specified version * Reboot for 1.3.0 * Notes that we could have different expect scripts * Put back in the restart after all of the domain updates * Scp uses ls1 instead of ls1.lme.local * Up the timeout of the adding ls1.lme.local * Up the timeout of the adding ls1.lme.local * Fixes chmod of the output.log for tests * Adds venv to the gitignore * Adds the ability to pass a branch to the installer * Remove node installer * Change timeout in expect script for slow connections * Make shell files executable --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> * Fix deploy.sh data retention failure error (#190) * Fix deploysh data retention failure (#179) * Update deploy.sh * Update deploy.sh * Update deploy.sh * Update deploy.sh * Remove free (#188) * changed the word free to no-cost or no-cost to users * rephrased wording to 'which comes at no cost to users' --------- Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update upgrading.md with data retention failure resolution (#189) --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Automatically Add Tags to Azure Resources (#186) * Add tags to all Azure resource creations calls --------- Co-authored-by: Clint Baxley <[email protected]> * Refactored long line and added switch for debug mode * Removed unnecessary comments * Update pull_request_template.md (#198) * Update pull_request_template.md Moved Squash commits from post-merge to pre-merge. * overriding default PR template for preferred LME template * overriding default PR template for preferred LME template * updating issue template to shorten the template --------- Co-authored-by: mreeve-snl <[email protected]> * Python testbed setup (#183) * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Hide pytest_cache * Add pycache to gitignore * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Create helpers and conftest for python tests * Setup for using test explorer in the dev environment * Adding azure shell requirements to docker image * Adding Python API tests * Merges additional tests * Made changes to fix tests that were failing * Separate linux only tests from others * Create a workflow for building test environments * Make the docker user be the same as the vbox user id * Set up to run the installer in docker * Pick up different fs types in data_retention * Change the build path for building lme container * Install lme after build * Make lme installer executable * Set up the build for tests * Add the cluster workflow for github actions --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Rishi <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#206) Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged. * Made changes to facilitate HTML Reports on test execution (#211) * Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting * Fixed Typos on Readme * Fixed Typos on Readme * removed tags flag from nsg because it was preventing some rules from being created (#214) Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#217) Using keywords like "fixes" or "closes" only auto-closes the corresponding issue if the PR is going to be merged into main. For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue. Added some documentation to clarify this. * Create new workflow for automating the release process (#199) --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: unknown <[email protected]> Co-authored-by: Grant (SNL) <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> Co-authored-by: Brown <[email protected]> Co-authored-by: mreeve-snl <[email protected]> Co-authored-by: Rishi <[email protected]> Co-authored-by: rishagg01 <[email protected]>
* update configelasticsearch function * Update deploy.sh * Update deploy.sh
* Add a markdown file for dashboard descriptions, link new file in README.md (#138) * Add a markdown file for dashboard descriptions ## 🗣 Description ## <!-- Describe the "what" of your changes in detail. --> Add dashboard-descriptions.md in /docs/markdown/reference. Add a link to this file within the main README.md's table of contents. ### 💭 Motivation and context <!-- Why is this change required? --> <!-- What problem does this change solve? How did you solve it? --> <!-- Mention any related issue(s) here using appropriate keywords such --> <!-- as "closes" or "resolves" to auto-close them on merge. --> The LME repository does not have a location for dashboard descriptions. ## 🧪 Testing <!-- How did you test your changes? How could someone else test this PR? --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> N/A ## ✅ Pre-approval checklist ## - [x] Changes are limited to a single goal **AND** the title reflects this in a clear human readable format - [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document. - [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR) - [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide). - [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR. ## ✅ Post-merge Checklist - [x] Squash all commits into one PR level commit - [x] Delete the branch to keep down number of branches * Update README.md to include dashboard-descriptions.md * Update wording for computer software overview dashboard * Fix some grammatical changes in dashboard-descriptions.md * Release 1.3.1 merge into main (#154) * Update retention function to fix retention policy bug (#143) * Updated troubleshooting guide to account for index management (#134) * Update upgrading.md to account for 1.3.1 (#151) * Update upgrading.md * Update upgrading.md --------- Co-authored-by: Andrew Arz <[email protected]> * Fixes dashboard update not importing on fresh install (#167) (#169) * Fixes dashboard update not importing on fresh install #165 * Update upgrading.md to include status on v1.3.2, along with revisions to the document overall * remove step 4 from upgrading.md; add additional instructions for v1.3.2 --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> * Add proof of concept selenium tests * Correct the script name in the doc string * User Security Selenium Tests for No Results Panels * First full selenium test. Currently just User Security * WIP User HR * Completed all dashboards. Requires testing now * Cut dev comments Co-authored-by: Alden Hilton <[email protected]> * Debugging a couple unit tests that error out. Two left * Install LME in the testbed from a single script (#150) * Adding the configure scripts * Add scripts to zip and copy to a container for downloading in the server * Grab the expiry time properly in copy file * Overwrite the blob if it exists * Add the script to download file into DC * Script that unzips the files in a container * Adds username argument to download files * Add script to run scripts in container * Adds username argument to gpo script * Modifies the url name in the client GPO * Adds the functionality for chapter 1 and first half of chapter 2 * Imports the sysmon GPO * Update the variables for sysmon gpo * Name the scripts so they are grouped together in a listing * Echos the file download url * Expands the domain name correctly in create ou * Write the url output of copy file to container to a different output stream * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Echos the file download url * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all of the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Adds InstallTestbed instructions to Readme.md * Modifies parameters to be pascal case * ls1 not being set on DC1 * Adds Linux Only install to SetupTestbed * Remove separate linux only script * Update testing/Readme.md Co-authored-by: Alden Hilton <[email protected]> * Make number of clients consisten between scripts * Add ports for elk stack for testing * Update readmes to change ResourceGroupName to ResourceGroup * Adds a switch to install linux only * Adds simple tests to check install * Removes the error if the old configure zip is not found. * Adds variables to linux tests run command * Move credential extraction to lib for use by other scripts. * Adds npm for other testing * Adds latest version of nodejs for testing * Make output.log readable for tests * Add the -m parameter in the testing readme * Download the latest version or a specified version * Reboot for 1.3.0 * Notes that we could have different expect scripts * Put back in the restart after all of the domain updates * Scp uses ls1 instead of ls1.lme.local * Up the timeout of the adding ls1.lme.local * Up the timeout of the adding ls1.lme.local * Fixes chmod of the output.log for tests * Adds venv to the gitignore * Adds the ability to pass a branch to the installer * Remove node installer * Change timeout in expect script for slow connections * Make shell files executable --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> * Fix deploy.sh data retention failure error (#190) * Fix deploysh data retention failure (#179) * Update deploy.sh * Update deploy.sh * Update deploy.sh * Update deploy.sh * Remove free (#188) * changed the word free to no-cost or no-cost to users * rephrased wording to 'which comes at no cost to users' --------- Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update upgrading.md with data retention failure resolution (#189) --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Automatically Add Tags to Azure Resources (#186) * Add tags to all Azure resource creations calls --------- Co-authored-by: Clint Baxley <[email protected]> * Switched script to headless mode * added switch for headless, detached, and debug mode. Bug where driver.quit does not close window. * Refactored long line and added switch for debug mode * Removed unnecessary comments * Update pull_request_template.md (#198) * Update pull_request_template.md Moved Squash commits from post-merge to pre-merge. * overriding default PR template for preferred LME template * overriding default PR template for preferred LME template * updating issue template to shorten the template --------- Co-authored-by: mreeve-snl <[email protected]> * Python testbed setup (#183) * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Hide pytest_cache * Add pycache to gitignore * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Create helpers and conftest for python tests * Setup for using test explorer in the dev environment * Adding azure shell requirements to docker image * Adding Python API tests * Merges additional tests * Made changes to fix tests that were failing * Separate linux only tests from others * Create a workflow for building test environments * Make the docker user be the same as the vbox user id * Set up to run the installer in docker * Pick up different fs types in data_retention * Change the build path for building lme container * Install lme after build * Make lme installer executable * Set up the build for tests * Add the cluster workflow for github actions --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Rishi <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#206) Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged. * Made changes to facilitate HTML Reports on test execution (#211) * Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting * Fixed Typos on Readme * Fixed Typos on Readme * removed tags flag from nsg because it was preventing some rules from being created (#214) Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#217) Using keywords like "fixes" or "closes" only auto-closes the corresponding issue if the PR is going to be merged into main. For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue. Added some documentation to clarify this. * Create new workflow for automating the release process (#199) * Update chapter4.md * Update bug-or-error-report.md Grammatical changes * Update feature_request.md grammatical changes * Update PULL_REQUEST_TEMPLATE.md punctuation changes * Update Readme.md grammatical changes * Update chapter2.md grammatical update * Update chapter4.md grammatical changes * Update prerequisites.md grammatical changes * Update CONTRIBUTING.md grammatical changes * Update README.md grammatical changes * Update RELEASES.md grammatical changes * Update Readme.md Grammtical changes * Update Readme.md more changes * Update README.md grammatical updates * Update README.md sentence updates * Update CONTRIBUTING.md changes * Update RELEASES.md comma changes * Update RELEASES.md grammar updates * Update Readme.md grammar changes * Update chapter4.md * Update faq.md * Update troubleshooting.md some small grammar changes * Update prerequisites.md verb changes * Update chapter1.md grammatical updates and changes * Update guide_to_ous.md updates * Update chapter3.md sentence structure changes and grammar updates * Update resilience.md removing passive voice * Update filtering.md sentence fixes * Update retention.md plural fixes * Update other-logging.md grammar changes * Update backups.md Grammatical changes * Update certificates.md grammar changes and sentence updates * Update upgrading.md sentence rewrites * added modifications per Dan Cernoch * small clarifications * small clarifications added * minor grammatical edits * Update prerequisites.md sentence changes * Update chapter4.md grammar updates and some changes * Update troubleshooting.md * Update CONTRIBUTING.md * Update README.md * Update Readme.md * Update chapter1.md * Update guide_to_ous.md * Update chapter3.md * Update upgrading.md grammar fixes * Update certificates.md removing passive voice * Update backups.md removing passive voice --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: unknown <[email protected]> Co-authored-by: Grant (SNL) <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> Co-authored-by: Brown <[email protected]> Co-authored-by: mreeve-snl <[email protected]> Co-authored-by: Rishi <[email protected]> Co-authored-by: rishagg01 <[email protected]> Co-authored-by: BDCISA <[email protected]> Co-authored-by: Christopher Morris <[email protected]>
# Conflicts: # .devcontainer/python_development/devcontainer.json # .github/ISSUE_TEMPLATE/bug-or-error-report.md # .github/PULL_REQUEST_TEMPLATE.md # .github/workflows/cluster.yml # .github/workflows/linux_only.yml # .gitignore # README.md # docs/markdown/chapter3/chapter3.md # docs/markdown/maintenance/upgrading.md # testing/InstallTestbed.ps1 # testing/Readme.md # testing/SetupTestbed.ps1 # testing/configure/lib/functions.sh # testing/configure/linux_install_lme.exp # testing/configure/linux_install_lme.sh # testing/development/Dockerfile # testing/development/build_docker_lme_install.sh # testing/development/docker-compose.yml # testing/tests/.env_example # testing/tests/README.md # testing/tests/api_tests/helpers.py # testing/tests/api_tests/winlogbeat/test_data/mapping_response.json # testing/tests/api_tests/winlogbeat/test_data/mapping_response_actual.json # testing/tests/api_tests/winlogbeat/test_data/winlog_search_data.json # testing/tests/api_tests/winlogbeat/test_server.py # testing/tests/requirements.txt
* Add a markdown file for dashboard descriptions, link new file in README.md (#138) * Add a markdown file for dashboard descriptions ## 🗣 Description ## <!-- Describe the "what" of your changes in detail. --> Add dashboard-descriptions.md in /docs/markdown/reference. Add a link to this file within the main README.md's table of contents. ### 💭 Motivation and context <!-- Why is this change required? --> <!-- What problem does this change solve? How did you solve it? --> <!-- Mention any related issue(s) here using appropriate keywords such --> <!-- as "closes" or "resolves" to auto-close them on merge. --> The LME repository does not have a location for dashboard descriptions. ## 🧪 Testing <!-- How did you test your changes? How could someone else test this PR? --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> N/A ## ✅ Pre-approval checklist ## - [x] Changes are limited to a single goal **AND** the title reflects this in a clear human readable format - [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document. - [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR) - [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide). - [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR. ## ✅ Post-merge Checklist - [x] Squash all commits into one PR level commit - [x] Delete the branch to keep down number of branches * Update README.md to include dashboard-descriptions.md * Update wording for computer software overview dashboard * Fix some grammatical changes in dashboard-descriptions.md * Release 1.3.1 merge into main (#154) * Update retention function to fix retention policy bug (#143) * Updated troubleshooting guide to account for index management (#134) * Update upgrading.md to account for 1.3.1 (#151) * Update upgrading.md * Update upgrading.md --------- Co-authored-by: Andrew Arz <[email protected]> * Fixes dashboard update not importing on fresh install (#167) (#169) * Fixes dashboard update not importing on fresh install #165 * Update upgrading.md to include status on v1.3.2, along with revisions to the document overall * remove step 4 from upgrading.md; add additional instructions for v1.3.2 --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> * Add proof of concept selenium tests * Correct the script name in the doc string * User Security Selenium Tests for No Results Panels * First full selenium test. Currently just User Security * WIP User HR * Completed all dashboards. Requires testing now * Cut dev comments Co-authored-by: Alden Hilton <[email protected]> * Debugging a couple unit tests that error out. Two left * Install LME in the testbed from a single script (#150) * Adding the configure scripts * Add scripts to zip and copy to a container for downloading in the server * Grab the expiry time properly in copy file * Overwrite the blob if it exists * Add the script to download file into DC * Script that unzips the files in a container * Adds username argument to download files * Add script to run scripts in container * Adds username argument to gpo script * Modifies the url name in the client GPO * Adds the functionality for chapter 1 and first half of chapter 2 * Imports the sysmon GPO * Update the variables for sysmon gpo * Name the scripts so they are grouped together in a listing * Echos the file download url * Expands the domain name correctly in create ou * Write the url output of copy file to container to a different output stream * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Echos the file download url * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all of the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Adds InstallTestbed instructions to Readme.md * Modifies parameters to be pascal case * ls1 not being set on DC1 * Adds Linux Only install to SetupTestbed * Remove separate linux only script * Update testing/Readme.md Co-authored-by: Alden Hilton <[email protected]> * Make number of clients consisten between scripts * Add ports for elk stack for testing * Update readmes to change ResourceGroupName to ResourceGroup * Adds a switch to install linux only * Adds simple tests to check install * Removes the error if the old configure zip is not found. * Adds variables to linux tests run command * Move credential extraction to lib for use by other scripts. * Adds npm for other testing * Adds latest version of nodejs for testing * Make output.log readable for tests * Add the -m parameter in the testing readme * Download the latest version or a specified version * Reboot for 1.3.0 * Notes that we could have different expect scripts * Put back in the restart after all of the domain updates * Scp uses ls1 instead of ls1.lme.local * Up the timeout of the adding ls1.lme.local * Up the timeout of the adding ls1.lme.local * Fixes chmod of the output.log for tests * Adds venv to the gitignore * Adds the ability to pass a branch to the installer * Remove node installer * Change timeout in expect script for slow connections * Make shell files executable --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> * Fix deploy.sh data retention failure error (#190) * Fix deploysh data retention failure (#179) * Update deploy.sh * Update deploy.sh * Update deploy.sh * Update deploy.sh * Remove free (#188) * changed the word free to no-cost or no-cost to users * rephrased wording to 'which comes at no cost to users' --------- Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update upgrading.md with data retention failure resolution (#189) --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Automatically Add Tags to Azure Resources (#186) * Add tags to all Azure resource creations calls --------- Co-authored-by: Clint Baxley <[email protected]> * Switched script to headless mode * added switch for headless, detached, and debug mode. Bug where driver.quit does not close window. * Refactored long line and added switch for debug mode * Removed unnecessary comments * Update pull_request_template.md (#198) * Update pull_request_template.md Moved Squash commits from post-merge to pre-merge. * overriding default PR template for preferred LME template * overriding default PR template for preferred LME template * updating issue template to shorten the template --------- Co-authored-by: mreeve-snl <[email protected]> * Python testbed setup (#183) * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Hide pytest_cache * Add pycache to gitignore * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Create helpers and conftest for python tests * Setup for using test explorer in the dev environment * Adding azure shell requirements to docker image * Adding Python API tests * Merges additional tests * Made changes to fix tests that were failing * Separate linux only tests from others * Create a workflow for building test environments * Make the docker user be the same as the vbox user id * Set up to run the installer in docker * Pick up different fs types in data_retention * Change the build path for building lme container * Install lme after build * Make lme installer executable * Set up the build for tests * Add the cluster workflow for github actions --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Rishi <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#206) Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged. * Made changes to facilitate HTML Reports on test execution (#211) * Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting * Fixed Typos on Readme * Fixed Typos on Readme * removed tags flag from nsg because it was preventing some rules from being created (#214) Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#217) Using keywords like "fixes" or "closes" only auto-closes the corresponding issue if the PR is going to be merged into main. For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue. Added some documentation to clarify this. * Create new workflow for automating the release process (#199) * Github workflows for building environments (#195) * Run the correct installer file * Run the installer from the root directory * Try a self hosted github runner * Reduce logging for docker pull. * Adds quiet flag to docker pull command * Pull the images before expect to reduce run time * Install docker early in order to speed up install * Builds the right docker-compose file * Increase timeout for linux install expect script * Change timeout on expect script * Change the way expect watches the script * Expand the timeout when waiting for Elasticsearch * Search for more output in the expect script * Change the match for the dots in expect * Change the regex for matching dots * Change the output for catching dots * Add chrome to Dockerfile for selenium * Import selenium tests and run python tests * Activate venv when running tests * Correct path for venv in the container * Correct path for venv in the container * Running only linux tests * Adjust scripts to run as a non super user * Change the permissions on the output log to source for environment variables later * Check for output log * Make output log available to test instantiation * Change pytest cache dir to home for user * Change pytest cache dir to home for user * Change pytest cache dir permissions * Hide get-docker.sh from installs * Cleanup test files in workflow * Add the cluster workflow for github actions * Adds a cluster build * Run the test cluster in pwsh * Fail pipeline when commands fail * Catch the error from powershell * Remove duplicate run command * Set env vars explicitly * Modify the escape char for env vars * Try a different method of catching errors in pwsh script * Check failure of pwsh script * Test successful run of build_cluster * Test failure of script * Capture the output from the az commands * Continue on error condition * Simplify run command * Try catching failures in a new way. * Test failure capture * Setting error action to continue * Remove ErrorAction * Use docker-compose run instead * Capture exit code to fail step * Try propigating errors from pwsh * Capture external command exit code * Send lastexitcode * Don't exit right away * Disable immediate stop on exit * Run simple test for exit code * Cd to docker compose file * Catch exec exit code * Remove unneded flags from the command * Adds back in the build script * Adds an explicit exit for powershell script * Remove spaces after escape character * Escape the exitcode variable in the shell command * Remove extra exit from build_cluster.ps1 * Add a passing command for build_cluster.ps1 * Move to the install directory * Run setup testbed to get an error * Try to build a cluster with the build_cluster.ps1 script. * Check resource group variable * Set the resource group name differently * Build a cluster using the generated resource group * Make the paths relative in the build_cluster script * Move to the right directory to do an install * Destroy cluster on pipeline finish * Change the owner of the files to match the host in the development container * Su user to remove testing files * Run the docker-compose as root to clean up * Run as root to clean up containers * Build the cluster in azure * List the files in the current directory on exec * Run the files from the new path * Investigate more about the file environment * Update the envornment for building the cluster * Update the environment users before docker up * Try to start hung job * List all the files with their owners in the container * Escape the powershell commands * Check the paths and files with bash * Find the path we are on * Check powershell environment * Cd to home directory in powershell * Cd to home directory in powershell * Rebuild docker compose as the right user * Change directory to source directory for powershell * Change to proper directory for powershell * Build a full cluster in pipeline * Run the linux tests and check permissions of files * Change permissions on output file with sudo * Turn off cluster creation for speed * Comment out building cluster in steps * Only delete the resource group if it exists * Adds ability to get the public ip for fw rules * Put the tags in quotes when creating nsg rules * Output the command being run for nsg rules * Remove tags for nsg port definitions * Install lme on the cluster * Builds the full cluster install * Cleans up the useage of the environment variables in pipeline * Extract environment variables from the build script and use them in the GitHub workflow. * Do a minimal linux install * Fix the path for retrieving env vars * Check setting of github env * Source the env file and push it to github env * Print some debug information to the console * Check setting of each key in functions * Parse the output for the passwords better * Uses a unique id instead of run_id to make sure it is unique * Double quote the file name for sed in output.log * Changes the way we get passwords from output.log * Make sure key doesn't have newline * Escape dollar sign * Properly escape double quotes inside of docker-compose command * Escape all of the dollar signs in the compose command * Write the environment variables to the githut environment * Clean up debugging output * Remove more debugging output * Remove set e * Adds function to write passwords to a file for actions * List files in directory after writing passwords * Export the env vars in the github file * Fail the workflow if the environment is not set correctly * Clean up the environment vars for the container * Set the variables on run of the pwsh command * Run commands on the domain controller * Get the envrionment checker to pass * Update passing variables to remote script * Escape the powershell environment variables * Change the case of the resource group env var * Don't destroy cluster so we can manually test * Build the entire cluster to run commands against * Run a command on the linux machine * Run remote tests * Run minimal installs to debug tests * Fix escaping for test commands * Move to the correct directory for tests * Add continuation characters to the lines in the script * Remove nested double quotes * Uses the ip of LS1 to run the tests on * Put the cluster build command on one line * Destroy clusters at the end * Quote output log correctly on build * Run all api tests on cluster * Build full cluster and add verbose logging to pytest * Stop deleting the cluster in the destroy_cluster.ps1 script * Modify installer to use the new winlogbeat index pattern * Try to get the dns to resolve ls1 * Add ls1 to the hosts file so it resolves always * Modify tests to pass on a working cluster * Skip the fragile test for mapping * Set up to run selenium tests on the cluster * Testing * Rerun build after rebasing to the right branch * Pass the minimal install flag to install lme * Build complete cluster and run all tests * Pull the images quietly if running without a terminal. * Run the simple tests on PR checkin and the longer ones when triggered * Build the linux docker container upon check in of a pr * Build lme container fresh before install * Runs an end to end build in docker and cluster * Print out the download feedback when pulling images * Build 1.4.0 branch * Build the cluster using the main branch of the repository * Allow passing branch to installers from the pipeline * Run tests from a different base branch * Remove the ampersand typo * Allow passing arguments to the installer scripts * Rearrange install arguments * Test passing arguments in install lme * Build lme without arguments * Install lme with no arguments * Run command as string in install_lme.ps1 * Build by passing arguments * Run a complete build using arguments * Update the sources to allow for updating in the pipeline * Build the cluster using the latest branch * Set up the latest branch var * Runs an upgrade in the pipeline * Run the upgrade in the remote linux machine * Run upgrade on minimal install * Checks out the current branch to run an upgrade on linux * Capture the exit code of the upgrade script * Check the directories we are working in * Clone the git repository to run the upgrade * Checkout the proper branch from origin * Get the remote username and home dir for the remote server * Set the home directory for the az user * Use origin when checking out in the upgrade script * Revert the changes to deploy.sh * Set a dumb terminal to avoid terminal errors * Export the terminal variable correctly * Capture the output of the upgrade script to fail pipeline if it fails * Revert previous changes as they seemed to break upgrade * Use a different format for executing the pwsh script * Destroy the cluster when done * Output the upgrade information to the terminal * Try capturing the docker-compose output * Directly capture the output of the compose command * Fixes unbalanced quote * Build and run full cluster with an upgrade * Builds the current brand for the cluster * Add a unique id for the docker-compose so you can run multiple instances of the same docker-compose file * Adds upgrade.yml to gh workflows * Runs both a build and an upgrade * Adds upgrade to the gh workflows * Get gh to notice new workflow * Match build names to parent branch * Trigger gh to see the workflow * Get gh actions to trigger workflow * Update code to get gh to see the actions * Update code to use the new workflow module. * Trigger gh actions to run * Get gh to run workflows * Try to get gh to run workflows * Change upgrade branch pulling * Checking out branch for upgrade in a new way * Rename workflow for upgrade * Convert to docker compose * Run all three builds using docker compose and -p * Clean up docker containers * Build the docker containers fresh for the linux_only workflow * Adds readme and checks an upgrade where the upgrade version is the same as the current version * Fixes typo in the workflow file * Runs docker as sudo * Remove the privileged flag from the lme container * Try leaving the swarm on the host if running in non privileged environment * Leave the swarm on the host * Reset to run docker as privileged * Installs the current branch in linux only * Stop pruning system to see if elastc starts faster * Don't take down the docker containers to see why they aren't working * Removes the gh actions shell escape vulnerability * Remove the docker containers at end of run * changing .github/README.md name to prevent it apperaing on main web page (#260) * Append the flags to the end of the password file (#263) * Append the flags to the end of the password file * Prints the contents of password.txt to the console * Extract the credentials in a new way to compensate for the flags being in the file * Tests a build that runs locally on github * Keep container running for debugging purposes * Fix the credentials parsing function * Create a workflow for a burndown chart (#302) * Display the chart in the burndown summary * Get workflow dispatch to show * Adds defaults for the burndown chart workflow * Clean up debugging information from the workflow (#310) * Clean up debugging information from the workflow * Increase column count to match the number of columns in the board. * Break up selenium tests (#281) * Adding selenium directory and readme * Separate out the selenium tests so they can be run separately * Run selenium tests in pipeline * Puts the variables for env one to a line * Issue # 289 selenium test for Computer Software Overview dashboard (#290) * Updated Selenium tests for Computer Overview Dashboard * Updated Selenium tests for Computer Overview Dashboard * Updated Selenium test scripts for Health Check Dashboard (#292) * Set up selenium tests to run on cluster test * Point tests to the proper test folder * Update Selenium tests for Process Explorer Dashboard (#295) * Rewrite completed for Selenium test scripts for Security Dashboard - Security Log (#300) * Rewrote Selenium Tests for Sysmon Summary Dashboard (#301) * Rewrite Selenium Tests for User HR Dashboard * Rewrite of Selenium Tests for User Security Dashboard (#304) --------- Co-authored-by: rishagg01 <[email protected]> Co-authored-by: Rishi <[email protected]> * API calls code for Data Insertion (#343) * modified: testing/tests/api_tests/helpers.py new file: testing/tests/api_tests/selenium_tests/__init__.py new file: testing/tests/api_tests/selenium_tests/conftest.py new file: testing/tests/api_tests/selenium_tests/fixtures/hosts.json new file: testing/tests/api_tests/selenium_tests/fixtures/logonevents.json new file: testing/tests/api_tests/selenium_tests/queries/filter_hosts.json new file: testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json new file: testing/tests/api_tests/selenium_tests/test_server.py * commit renamed: testing/tests/api_tests/selenium_tests/__init__.py -> testing/tests/api_tests/data_insertion_tests/__init__.py commit renamed: testing/tests/api_tests/selenium_tests/conftest.py -> testing/tests/api_tests/data_insertion_tests/conftest.py commit renamed: testing/tests/api_tests/selenium_tests/fixtures/hosts.json -> testing/tests/api_tests/data_insertion_tests/fixtures/hosts.json commit renamed: testing/tests/api_tests/selenium_tests/fixtures/logonevents.json -> testing/tests/api_tests/data_insertion_tests/fixtures/logonevents.json commit renamed: testing/tests/api_tests/selenium_tests/queries/filter_hosts.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_hosts.json commit renamed: testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_logonevents.json commit renamed: testing/tests/api_tests/selenium_tests/test_server.py -> testing/tests/api_tests/data_insertion_tests/test_server.py commit modified: testing/tests/api_tests/helpers.py * Updated selenium tests for USER HR dashboard panels post data insertion (#358) * Updated User HR Dashboard Selenium Test for User HR Logon Title panel (#385) * Updated selenium tests for USER HR dashboard panels post data insertion * Updated User HR Dashboard Selenium Test for User HR Logon Title panel * Adds some extra files to gitignore * 1.4gitignore * Skip the non passing test * Skip failing tests * Comment out failing tests * Update the merging version --------- Co-authored-by: mitchelbaker-cisa <[email protected]> Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: unknown <[email protected]> Co-authored-by: Grant (SNL) <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> Co-authored-by: Brown <[email protected]> Co-authored-by: mreeve-snl <[email protected]> Co-authored-by: Rishi <[email protected]> Co-authored-by: rishagg01 <[email protected]> Co-authored-by: Connor <[email protected]>
* New wec config xml for ad id logging and new dashboards * updated zip to contain new GPOs * Divine's dashboards * Fixed bug in lme_wec_config, missing asterisk in queries * Fixed casing on Query ID, must be Id * Fixed typo empty path string * Fixed scheduler typo * Renamed dashboards to fit convention * Dashboard descriptions * Fixed spacing * Fixed spacing 2 * Adjusted dashboard names * Typo fix * Upgrade instructions for release 1.4.0 for ID Logging * removing text * Adds code to upgrade to 1.4.0 * Check the latest and version in the upgrade section * Adds a better check for upgrading from 1.3.x to 1.4.x * Modify the version check for upgrades * Updates the dashboards to be all exported by the exporter script * Adds the updated menu to all of the dashboards * Update the upgrade directions for those who downloaded the version --------- Co-authored-by: Brown <rgbrow> Co-authored-by: unknown <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: cbaxley <[email protected]>
mitchelbaker-cisa
requested review from
cbaxley,
rgbrow1949,
ddiabe,
mreeve-snl and
aarz-snl
ddiabe
approved these changes
Aug 21, 2024
# Conflicts: # .gitignore # Chapter 3 Files/deploy.sh # testing/development/upgrade_lme.sh # testing/merging_version.sh # testing/tests/api_tests/data_insertion_tests/test_server.py # testing/tests/api_tests/helpers.py # testing/tests/selenium_tests/cluster/test_security_dashboard_security_log.py
* Add a markdown file for dashboard descriptions, link new file in README.md (#138) * Add a markdown file for dashboard descriptions ## 🗣 Description ## <!-- Describe the "what" of your changes in detail. --> Add dashboard-descriptions.md in /docs/markdown/reference. Add a link to this file within the main README.md's table of contents. ### 💭 Motivation and context <!-- Why is this change required? --> <!-- What problem does this change solve? How did you solve it? --> <!-- Mention any related issue(s) here using appropriate keywords such --> <!-- as "closes" or "resolves" to auto-close them on merge. --> The LME repository does not have a location for dashboard descriptions. ## 🧪 Testing <!-- How did you test your changes? How could someone else test this PR? --> <!-- Include details of your testing environment, and the tests you ran to --> <!-- see how your change affects other areas of the code, etc. --> N/A ## ✅ Pre-approval checklist ## - [x] Changes are limited to a single goal **AND** the title reflects this in a clear human readable format - [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document. - [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR) - [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide). - [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR. ## ✅ Post-merge Checklist - [x] Squash all commits into one PR level commit - [x] Delete the branch to keep down number of branches * Update README.md to include dashboard-descriptions.md * Update wording for computer software overview dashboard * Fix some grammatical changes in dashboard-descriptions.md * Release 1.3.1 merge into main (#154) * Update retention function to fix retention policy bug (#143) * Updated troubleshooting guide to account for index management (#134) * Update upgrading.md to account for 1.3.1 (#151) * Update upgrading.md * Update upgrading.md --------- Co-authored-by: Andrew Arz <[email protected]> * Fixes dashboard update not importing on fresh install (#167) (#169) * Fixes dashboard update not importing on fresh install #165 * Update upgrading.md to include status on v1.3.2, along with revisions to the document overall * remove step 4 from upgrading.md; add additional instructions for v1.3.2 --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> * Add proof of concept selenium tests * Correct the script name in the doc string * User Security Selenium Tests for No Results Panels * First full selenium test. Currently just User Security * WIP User HR * Completed all dashboards. Requires testing now * Cut dev comments Co-authored-by: Alden Hilton <[email protected]> * Debugging a couple unit tests that error out. Two left * Install LME in the testbed from a single script (#150) * Adding the configure scripts * Add scripts to zip and copy to a container for downloading in the server * Grab the expiry time properly in copy file * Overwrite the blob if it exists * Add the script to download file into DC * Script that unzips the files in a container * Adds username argument to download files * Add script to run scripts in container * Adds username argument to gpo script * Modifies the url name in the client GPO * Adds the functionality for chapter 1 and first half of chapter 2 * Imports the sysmon GPO * Update the variables for sysmon gpo * Name the scripts so they are grouped together in a listing * Echos the file download url * Expands the domain name correctly in create ou * Write the url output of copy file to container to a different output stream * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Echos the file download url * Create a new LME folder for our scripts and files * Set path for extract to lme * Update paths for scripts to /lme * Update paths for scripts to /lme * Fix the wec server name setting * Adds the scripts to install chapter 1 and 2 * Allows azure to download in linux and windows * Adds linux install scripts. * Adds winlogbeat installer * emove garbage in update server name * Tweak several scripts to get the scp of files_for_windows * Adds installer script to run all of the scripts * Fixes the formatting method for az output * Clean up the scripts and add documentation * Fixes outputting format errors * Fixes hanging on adding ls1 to domain * Fix formatting errors on responses * Update linux expect script for different prompts. * Handle the reboot message for linux expect script * Adds InstallTestbed instructions to Readme.md * Modifies parameters to be pascal case * ls1 not being set on DC1 * Adds Linux Only install to SetupTestbed * Remove separate linux only script * Update testing/Readme.md Co-authored-by: Alden Hilton <[email protected]> * Make number of clients consisten between scripts * Add ports for elk stack for testing * Update readmes to change ResourceGroupName to ResourceGroup * Adds a switch to install linux only * Adds simple tests to check install * Removes the error if the old configure zip is not found. * Adds variables to linux tests run command * Move credential extraction to lib for use by other scripts. * Adds npm for other testing * Adds latest version of nodejs for testing * Make output.log readable for tests * Add the -m parameter in the testing readme * Download the latest version or a specified version * Reboot for 1.3.0 * Notes that we could have different expect scripts * Put back in the restart after all of the domain updates * Scp uses ls1 instead of ls1.lme.local * Up the timeout of the adding ls1.lme.local * Up the timeout of the adding ls1.lme.local * Fixes chmod of the output.log for tests * Adds venv to the gitignore * Adds the ability to pass a branch to the installer * Remove node installer * Change timeout in expect script for slow connections * Make shell files executable --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> * Fix deploy.sh data retention failure error (#190) * Fix deploysh data retention failure (#179) * Update deploy.sh * Update deploy.sh * Update deploy.sh * Update deploy.sh * Remove free (#188) * changed the word free to no-cost or no-cost to users * rephrased wording to 'which comes at no cost to users' --------- Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update upgrading.md with data retention failure resolution (#189) --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Automatically Add Tags to Azure Resources (#186) * Add tags to all Azure resource creations calls --------- Co-authored-by: Clint Baxley <[email protected]> * Switched script to headless mode * added switch for headless, detached, and debug mode. Bug where driver.quit does not close window. * Refactored long line and added switch for debug mode * Removed unnecessary comments * Update pull_request_template.md (#198) * Update pull_request_template.md Moved Squash commits from post-merge to pre-merge. * overriding default PR template for preferred LME template * overriding default PR template for preferred LME template * updating issue template to shorten the template --------- Co-authored-by: mreeve-snl <[email protected]> * Python testbed setup (#183) * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Hide pytest_cache * Add pycache to gitignore * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Add simple tests for http requests * Add an env file to gitignore * Remove unneeded pip install * Adds dev containers for vscode * Adds testing information for vscode * Uses .env file for tests if present * Adds env example file * Modify development container name * Adds readme for the testing environment * Create helpers and conftest for python tests * Setup for using test explorer in the dev environment * Adding azure shell requirements to docker image * Adding Python API tests * Merges additional tests * Made changes to fix tests that were failing * Separate linux only tests from others * Create a workflow for building test environments * Make the docker user be the same as the vbox user id * Set up to run the installer in docker * Pick up different fs types in data_retention * Change the build path for building lme container * Install lme after build * Make lme installer executable * Set up the build for tests * Add the cluster workflow for github actions --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Rishi <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#206) Added instruction to select Issue in Development area so that the corresponding Issue is automatically closed when the PR is merged. * Made changes to facilitate HTML Reports on test execution (#211) * Made changes to requirements.txt, ReadMe and gitignore to facilitate HTML reporting * Fixed Typos on Readme * Fixed Typos on Readme * removed tags flag from nsg because it was preventing some rules from being created (#214) Co-authored-by: Linda Lovero-Waterhouse <[email protected]> * Update PULL_REQUEST_TEMPLATE.md (#217) Using keywords like "fixes" or "closes" only auto-closes the corresponding issue if the PR is going to be merged into main. For PR's merged into release branches, we need to add the issue to the development box in the right sidebar in order to auto-close the issue. Added some documentation to clarify this. * Create new workflow for automating the release process (#199) * Github workflows for building environments (#195) * Run the correct installer file * Run the installer from the root directory * Try a self hosted github runner * Reduce logging for docker pull. * Adds quiet flag to docker pull command * Pull the images before expect to reduce run time * Install docker early in order to speed up install * Builds the right docker-compose file * Increase timeout for linux install expect script * Change timeout on expect script * Change the way expect watches the script * Expand the timeout when waiting for Elasticsearch * Search for more output in the expect script * Change the match for the dots in expect * Change the regex for matching dots * Change the output for catching dots * Add chrome to Dockerfile for selenium * Import selenium tests and run python tests * Activate venv when running tests * Correct path for venv in the container * Correct path for venv in the container * Running only linux tests * Adjust scripts to run as a non super user * Change the permissions on the output log to source for environment variables later * Check for output log * Make output log available to test instantiation * Change pytest cache dir to home for user * Change pytest cache dir to home for user * Change pytest cache dir permissions * Hide get-docker.sh from installs * Cleanup test files in workflow * Add the cluster workflow for github actions * Adds a cluster build * Run the test cluster in pwsh * Fail pipeline when commands fail * Catch the error from powershell * Remove duplicate run command * Set env vars explicitly * Modify the escape char for env vars * Try a different method of catching errors in pwsh script * Check failure of pwsh script * Test successful run of build_cluster * Test failure of script * Capture the output from the az commands * Continue on error condition * Simplify run command * Try catching failures in a new way. * Test failure capture * Setting error action to continue * Remove ErrorAction * Use docker-compose run instead * Capture exit code to fail step * Try propigating errors from pwsh * Capture external command exit code * Send lastexitcode * Don't exit right away * Disable immediate stop on exit * Run simple test for exit code * Cd to docker compose file * Catch exec exit code * Remove unneded flags from the command * Adds back in the build script * Adds an explicit exit for powershell script * Remove spaces after escape character * Escape the exitcode variable in the shell command * Remove extra exit from build_cluster.ps1 * Add a passing command for build_cluster.ps1 * Move to the install directory * Run setup testbed to get an error * Try to build a cluster with the build_cluster.ps1 script. * Check resource group variable * Set the resource group name differently * Build a cluster using the generated resource group * Make the paths relative in the build_cluster script * Move to the right directory to do an install * Destroy cluster on pipeline finish * Change the owner of the files to match the host in the development container * Su user to remove testing files * Run the docker-compose as root to clean up * Run as root to clean up containers * Build the cluster in azure * List the files in the current directory on exec * Run the files from the new path * Investigate more about the file environment * Update the envornment for building the cluster * Update the environment users before docker up * Try to start hung job * List all the files with their owners in the container * Escape the powershell commands * Check the paths and files with bash * Find the path we are on * Check powershell environment * Cd to home directory in powershell * Cd to home directory in powershell * Rebuild docker compose as the right user * Change directory to source directory for powershell * Change to proper directory for powershell * Build a full cluster in pipeline * Run the linux tests and check permissions of files * Change permissions on output file with sudo * Turn off cluster creation for speed * Comment out building cluster in steps * Only delete the resource group if it exists * Adds ability to get the public ip for fw rules * Put the tags in quotes when creating nsg rules * Output the command being run for nsg rules * Remove tags for nsg port definitions * Install lme on the cluster * Builds the full cluster install * Cleans up the useage of the environment variables in pipeline * Extract environment variables from the build script and use them in the GitHub workflow. * Do a minimal linux install * Fix the path for retrieving env vars * Check setting of github env * Source the env file and push it to github env * Print some debug information to the console * Check setting of each key in functions * Parse the output for the passwords better * Uses a unique id instead of run_id to make sure it is unique * Double quote the file name for sed in output.log * Changes the way we get passwords from output.log * Make sure key doesn't have newline * Escape dollar sign * Properly escape double quotes inside of docker-compose command * Escape all of the dollar signs in the compose command * Write the environment variables to the githut environment * Clean up debugging output * Remove more debugging output * Remove set e * Adds function to write passwords to a file for actions * List files in directory after writing passwords * Export the env vars in the github file * Fail the workflow if the environment is not set correctly * Clean up the environment vars for the container * Set the variables on run of the pwsh command * Run commands on the domain controller * Get the envrionment checker to pass * Update passing variables to remote script * Escape the powershell environment variables * Change the case of the resource group env var * Don't destroy cluster so we can manually test * Build the entire cluster to run commands against * Run a command on the linux machine * Run remote tests * Run minimal installs to debug tests * Fix escaping for test commands * Move to the correct directory for tests * Add continuation characters to the lines in the script * Remove nested double quotes * Uses the ip of LS1 to run the tests on * Put the cluster build command on one line * Destroy clusters at the end * Quote output log correctly on build * Run all api tests on cluster * Build full cluster and add verbose logging to pytest * Stop deleting the cluster in the destroy_cluster.ps1 script * Modify installer to use the new winlogbeat index pattern * Try to get the dns to resolve ls1 * Add ls1 to the hosts file so it resolves always * Modify tests to pass on a working cluster * Skip the fragile test for mapping * Set up to run selenium tests on the cluster * Testing * Rerun build after rebasing to the right branch * Pass the minimal install flag to install lme * Build complete cluster and run all tests * Pull the images quietly if running without a terminal. * Run the simple tests on PR checkin and the longer ones when triggered * Build the linux docker container upon check in of a pr * Build lme container fresh before install * Runs an end to end build in docker and cluster * Print out the download feedback when pulling images * Build 1.4.0 branch * Build the cluster using the main branch of the repository * Allow passing branch to installers from the pipeline * Run tests from a different base branch * Remove the ampersand typo * Allow passing arguments to the installer scripts * Rearrange install arguments * Test passing arguments in install lme * Build lme without arguments * Install lme with no arguments * Run command as string in install_lme.ps1 * Build by passing arguments * Run a complete build using arguments * Update the sources to allow for updating in the pipeline * Build the cluster using the latest branch * Set up the latest branch var * Runs an upgrade in the pipeline * Run the upgrade in the remote linux machine * Run upgrade on minimal install * Checks out the current branch to run an upgrade on linux * Capture the exit code of the upgrade script * Check the directories we are working in * Clone the git repository to run the upgrade * Checkout the proper branch from origin * Get the remote username and home dir for the remote server * Set the home directory for the az user * Use origin when checking out in the upgrade script * Revert the changes to deploy.sh * Set a dumb terminal to avoid terminal errors * Export the terminal variable correctly * Capture the output of the upgrade script to fail pipeline if it fails * Revert previous changes as they seemed to break upgrade * Use a different format for executing the pwsh script * Destroy the cluster when done * Output the upgrade information to the terminal * Try capturing the docker-compose output * Directly capture the output of the compose command * Fixes unbalanced quote * Build and run full cluster with an upgrade * Builds the current brand for the cluster * Add a unique id for the docker-compose so you can run multiple instances of the same docker-compose file * Adds upgrade.yml to gh workflows * Runs both a build and an upgrade * Adds upgrade to the gh workflows * Get gh to notice new workflow * Match build names to parent branch * Trigger gh to see the workflow * Get gh actions to trigger workflow * Update code to get gh to see the actions * Update code to use the new workflow module. * Trigger gh actions to run * Get gh to run workflows * Try to get gh to run workflows * Change upgrade branch pulling * Checking out branch for upgrade in a new way * Rename workflow for upgrade * Convert to docker compose * Run all three builds using docker compose and -p * Clean up docker containers * Build the docker containers fresh for the linux_only workflow * Adds readme and checks an upgrade where the upgrade version is the same as the current version * Fixes typo in the workflow file * Runs docker as sudo * Remove the privileged flag from the lme container * Try leaving the swarm on the host if running in non privileged environment * Leave the swarm on the host * Reset to run docker as privileged * Installs the current branch in linux only * Stop pruning system to see if elastc starts faster * Don't take down the docker containers to see why they aren't working * Removes the gh actions shell escape vulnerability * Remove the docker containers at end of run * changing .github/README.md name to prevent it apperaing on main web page (#260) * Append the flags to the end of the password file (#263) * Append the flags to the end of the password file * Prints the contents of password.txt to the console * Extract the credentials in a new way to compensate for the flags being in the file * Tests a build that runs locally on github * Keep container running for debugging purposes * Fix the credentials parsing function * Create a workflow for a burndown chart (#302) * Display the chart in the burndown summary * Get workflow dispatch to show * Adds defaults for the burndown chart workflow * Clean up debugging information from the workflow (#310) * Clean up debugging information from the workflow * Increase column count to match the number of columns in the board. * Break up selenium tests (#281) * Adding selenium directory and readme * Separate out the selenium tests so they can be run separately * Run selenium tests in pipeline * Puts the variables for env one to a line * Issue # 289 selenium test for Computer Software Overview dashboard (#290) * Updated Selenium tests for Computer Overview Dashboard * Updated Selenium tests for Computer Overview Dashboard * Updated Selenium test scripts for Health Check Dashboard (#292) * Set up selenium tests to run on cluster test * Point tests to the proper test folder * Update Selenium tests for Process Explorer Dashboard (#295) * Rewrite completed for Selenium test scripts for Security Dashboard - Security Log (#300) * Rewrote Selenium Tests for Sysmon Summary Dashboard (#301) * Rewrite Selenium Tests for User HR Dashboard * Rewrite of Selenium Tests for User Security Dashboard (#304) --------- Co-authored-by: rishagg01 <[email protected]> Co-authored-by: Rishi <[email protected]> * API calls code for Data Insertion (#343) * modified: testing/tests/api_tests/helpers.py new file: testing/tests/api_tests/selenium_tests/__init__.py new file: testing/tests/api_tests/selenium_tests/conftest.py new file: testing/tests/api_tests/selenium_tests/fixtures/hosts.json new file: testing/tests/api_tests/selenium_tests/fixtures/logonevents.json new file: testing/tests/api_tests/selenium_tests/queries/filter_hosts.json new file: testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json new file: testing/tests/api_tests/selenium_tests/test_server.py * commit renamed: testing/tests/api_tests/selenium_tests/__init__.py -> testing/tests/api_tests/data_insertion_tests/__init__.py commit renamed: testing/tests/api_tests/selenium_tests/conftest.py -> testing/tests/api_tests/data_insertion_tests/conftest.py commit renamed: testing/tests/api_tests/selenium_tests/fixtures/hosts.json -> testing/tests/api_tests/data_insertion_tests/fixtures/hosts.json commit renamed: testing/tests/api_tests/selenium_tests/fixtures/logonevents.json -> testing/tests/api_tests/data_insertion_tests/fixtures/logonevents.json commit renamed: testing/tests/api_tests/selenium_tests/queries/filter_hosts.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_hosts.json commit renamed: testing/tests/api_tests/selenium_tests/queries/filter_logonevents.json -> testing/tests/api_tests/data_insertion_tests/queries/filter_logonevents.json commit renamed: testing/tests/api_tests/selenium_tests/test_server.py -> testing/tests/api_tests/data_insertion_tests/test_server.py commit modified: testing/tests/api_tests/helpers.py * Updated selenium tests for USER HR dashboard panels post data insertion (#358) * Updated User HR Dashboard Selenium Test for User HR Logon Title panel (#385) * Updated selenium tests for USER HR dashboard panels post data insertion * Updated User HR Dashboard Selenium Test for User HR Logon Title panel * Added test data for Process Explorer - Downloads panel and updated selenium test for the panel (#395) * Updated API Data Insert test for filter host (#400) * Added test data for Process Explorer - Downloads panel and updated selenium test for the panel * Updated API Data Insert test for filter host * insert time delays in data insert function * Updated API & Sel tests for suspiscious powershell panel (#405) * Updated API & Sel tests for Create Remote Threads panel (#408) * Updated API & Sel tests for Powershell Network Connections panel on User Security Dashboard (#415) * Updated API & Sel tests for Create Remote Threads panel * Updated tests for Powershell network connections panel --------- Co-authored-by: mitchelbaker-cisa <[email protected]> Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: unknown <[email protected]> Co-authored-by: Grant (SNL) <[email protected]> Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: Linda Waterhouse <[email protected]> Co-authored-by: Linda Lovero-Waterhouse <[email protected]> Co-authored-by: Brown <[email protected]> Co-authored-by: mreeve-snl <[email protected]> Co-authored-by: Rishi <[email protected]> Co-authored-by: rishagg01 <[email protected]> Co-authored-by: Connor <[email protected]>
cbaxley
approved these changes
Aug 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗣 Description
Create merge commit for release 1.4.0.
✅ Pre-approval checklist
the title reflects this in a clear human readable format
✅ Pre-merge Checklist
✅ Post-merge Checklist