Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup redundant data after migrating vanilla K8s tests #2063

Merged
merged 9 commits into from
Mar 28, 2024
Merged

Cleanup redundant data after migrating vanilla K8s tests #2063

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a9d116e
cleanup old k8s tests
gurevichdmitry Mar 26, 2024
792a2ea
cleanup common lib
gurevichdmitry Mar 26, 2024
37bae1e
Merge branch 'main' into dg-cleanup-k8s-tests
gurevichdmitry Mar 27, 2024
6e470d2
Merge branch 'main' into dg-cleanup-k8s-tests
gurevichdmitry Mar 27, 2024
25702ba
add tmate session to debug eks-ci
gurevichdmitry Mar 27, 2024
6e45d83
Merge branch 'dg-cleanup-k8s-tests' of github.com:elastic/cloudbeat i…
gurevichdmitry Mar 27, 2024
0738241
add tmate session correct location
gurevichdmitry Mar 27, 2024
20621c4
remove tmate session
gurevichdmitry Mar 27, 2024
ff6788b
set env elk_version
gurevichdmitry Mar 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/eks-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,8 @@ jobs:
- name: Deploy tests Helm chart
id: deploy_helm
if: success()
env:
ELK_VERSION: 8.13.0-SNAPSHOT
run: |
just deploy-tests-helm ${{ matrix.test-target }} ${{ matrix.values_file }} ${{ matrix.range }}

Expand Down
217 changes: 0 additions & 217 deletions .github/workflows/k8s-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,220 +206,3 @@ jobs:
id: clear_cache
if: always()
run: rm -rf /tmp/*-${{ env.CONTAINER_SUFFIX }}.tar

k8s_functional_tests_full:
# Run full test suit on post-merge
name: ${{ matrix.test-target }}-${{ matrix.range }}-tests
if: github.event_name == 'push'
needs: [ Build ]
runs-on: ubuntu-22.04
timeout-minutes: 55
env:
KIND_CONFIG: kind-multi
strategy:
fail-fast: false
matrix:
include:
- test-target: pre_merge_agent
range: ''
values_file: tests/test_environments/values/ci-sa-agent.yml
- test-target: pre_merge
range: ""
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "0..5"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "5..10"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "10..15"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "15..20"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "20..25"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "25..30"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "30..35"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "35..40"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "40..45"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "45..50"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "50..55"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "55..60"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "60..65"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "65..70"
values_file: tests/test_environments/values/ci.yml
- test-target: file_system_rules
range: "70.."
values_file: tests/test_environments/values/ci.yml
- test-target: k8s_object_rules
range: "0..6"
values_file: tests/test_environments/values/ci.yml
- test-target: k8s_object_rules
range: "6..12"
values_file: tests/test_environments/values/ci.yml
- test-target: k8s_object_rules
range: "12..18"
values_file: tests/test_environments/values/ci.yml
- test-target: k8s_object_rules
range: "18.."
values_file: tests/test_environments/values/ci.yml
- test-target: process_scheduler_rules
range: "0..3"
values_file: tests/test_environments/values/ci.yml
- test-target: process_scheduler_rules
range: "3.."
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "0..5"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "5..10"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "10..15"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "15..20"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "20..24"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "24..28"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "28..32"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "32..36"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "36..40"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "40..44"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "44..48"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "48..52"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "52..56"
values_file: tests/test_environments/values/ci.yml
- test-target: process_api_server_rules
range: "56.."
values_file: tests/test_environments/values/ci.yml
- test-target: process_controller_manager_rules
range: "0..4"
values_file: tests/test_environments/values/ci.yml
- test-target: process_controller_manager_rules
range: "4..8"
values_file: tests/test_environments/values/ci.yml
- test-target: process_controller_manager_rules
range: "8..12"
values_file: tests/test_environments/values/ci.yml
- test-target: process_controller_manager_rules
range: "12.."
values_file: tests/test_environments/values/ci.yml
- test-target: process_etcd_rules
range: "0..4"
values_file: tests/test_environments/values/ci.yml
- test-target: process_etcd_rules
range: "4..8"
values_file: tests/test_environments/values/ci.yml
- test-target: process_etcd_rules
range: "8.."
values_file: tests/test_environments/values/ci.yml
- test-target: process_kubelet_rules
range: "0..4"
values_file: tests/test_environments/values/ci.yml
- test-target: process_kubelet_rules
range: "4..8"
values_file: tests/test_environments/values/ci.yml
- test-target: process_kubelet_rules
range: "8..12"
values_file: tests/test_environments/values/ci.yml
- test-target: process_kubelet_rules
range: "12..16"
values_file: tests/test_environments/values/ci.yml
- test-target: process_kubelet_rules
range: "16..20"
values_file: tests/test_environments/values/ci.yml
- test-target: process_kubelet_rules
range: "20.."
values_file: tests/test_environments/values/ci.yml
steps:
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
android: true
dotnet: true
haskell: true
large-packages: false
docker-images: true
swap-storage: true
- name: Check out the repo
uses: actions/checkout@v4

- name: Init Hermit
run: ./bin/hermit env -r >> $GITHUB_ENV

- name: Create k8s Kind Cluster
run: |
just create-kind-cluster ${{ env.KIND_CONFIG }}

- name: Cache docker images
uses: actions/cache@v4
with:
path: /tmp/*.tar
key: ${{ runner.os }}-dockers-cache-${{ env.CONTAINER_SUFFIX }}

- name: Load images to kind
run: |
./.ci/scripts/kind-images.sh ${{ env.CONTAINER_SUFFIX }} ${{ env.KIND_CONFIG }}
shell: bash

- name: Deploy tests Helm chart
id: deploy_helm
run: |
just deploy-tests-helm '${{ matrix.test-target }}' ${{ matrix.values_file }} ${{ matrix.range }}

- name: Run Tests
id: run_tests
run: |
just run-tests '${{ matrix.test-target }}' kind-${{ env.KIND_CONFIG }}

- name: Upload Test Results
if: always()
uses: actions/upload-artifact@v3
with:
name: allure_results-${{ matrix.test-target}}
path: tests/allure/results/

- name: clear cache
id: clear_cache
if: always()
run: rm -rf /tmp/*-${{ env.CONTAINER_SUFFIX }}.tar
97 changes: 0 additions & 97 deletions tests/commonlib/io_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,103 +217,6 @@ def in_place_copy(source, destination):
for line in sfile:
dfile.write(line)

@staticmethod
def edit_process_file(container_name: str, dictionary, resource: str):
"""
This function edits a process file
@param container_name: Container node
@param dictionary: Process parameters to set/unset
@param resource: File / Resource path
@return: None
"""
if container_name == "":
raise ValueError("Unknown container name is sent")

current_resource = Path(resource)
if not current_resource.is_file():
raise FileNotFoundError(f"File {resource} does not exist or mount missing.")

# Open and load the YAML into variable
with current_resource.open(encoding="utf-8") as file:
r_file = yaml.safe_load(file)

# Get process configuration arguments
arguments = r_file["spec"]["containers"][0]["command"]

# Collect set/unset keys and values from the dictionary
set_dict = dictionary.get("set", {})
unset_list = dictionary.get("unset", [])

# Cycle across set items from the dictionary
for s_key, s_value in set_dict.items():
# Find if set key exists already in the configuration arguments
if any(s_key == x.split("=")[0] for x in arguments):
# Replace the value of the key with the new value from the set items
arguments = [f"{s_key}={s_value}" if arg.split("=")[0] == s_key else arg for arg in arguments]
else:
# In case of non-existing key in the configuration arguments,
# append the key/value from set items
arguments.append(f"{s_key}={s_value}")

# Cycle across unset items from the dictionary
for us_key in unset_list:
# Filter out the unset keys from the configuration arguments
arguments = [x for x in arguments if us_key != x.split("=")[0]]

# Override the configuration arguments with the newly built configuration arguments
r_file["spec"]["containers"][0]["command"] = arguments

# Write the newly built configuration arguments
with current_resource.open(mode="w", encoding="utf-8") as file:
yaml.dump(r_file, file)

@staticmethod
def edit_config_file(container_name: str, dictionary, resource: str):
"""
This function edits a config file
@param container_name: Container node
@param dictionary: Config parameters to set/unset
@param resource: Config path
@return: None
"""
if container_name == "":
raise ValueError("Unknown container name is sent")

current_resource = Path(resource)
if not current_resource.is_file():
raise FileExistsError(f"File {resource} does not exist or mount missing.")

# Open and load the YAML into variable
with current_resource.open(encoding="utf-8") as file:
r_file = yaml.safe_load(file)

# Collect set/unset keys and values from the dictionary
set_dict = dictionary.get("set", {})
unset_list = dictionary.get("unset", [])

# Merge two dictionaries with priority for the set items
r_file = {**r_file, **set_dict}

# Cycle across unset items from the dictionary
for us_key in unset_list:
# Parsed dot separated key values
keys = us_key.split(".")
key_to_del = keys.pop()
r_dict = r_file

# Advance inside the dictionary for nested keys
for key in keys:
r_dict = r_dict.get(key, None)
if r_dict is None:
# Non-existing nested key
break
# Remove nested keys when all path exists
if r_dict:
del r_dict[key_to_del]
# Write the newly built config
with current_resource.open(mode="w", encoding="utf-8") as file:
yaml.dump(r_file, file)

@staticmethod
def get_beat_status_from_json(response: str, beat_name: str) -> str:
"""
Expand Down
Loading
Loading