[Refactor] E2E tests stracture

.github/workflows/tests.yaml

@@ -0,0 +1,44 @@
+name: Tests
+on: [pull_request]
+jobs:
+  unit-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Setup Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: |
+          pip install -r requirements.txt
+          pip install responses
+          pip install pytest
+          pip install pytest-cov
+      - name: Run tests
+        run: PYTHONPATH=. pytest --cov-report term-missing --cov=src tests/UnitTests/test_*.py
+
+  e2e-tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Setup Python 3.12
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: |
+          pip install -r requirements.txt
+          pip install docker
+          pip install requests
+      - name: Run E2E tests
+        # Set required environment variables
+        env:
+          LOGZIO_API_TOKEN: ${{ secrets.LOGZIO_API_TOKEN }}
+          DOCKERHUB_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
+          DOCKERHUB_USER: ${{ secrets.DOCKER_USERNAME }}
+          LOGZIO_SHIPPING_TOKEN: ${{ secrets.LOGZIO_SHIPPING_TOKEN }}
+          AZURE_AD_TENANT_ID: ${{ secrets.AZURE_AD_TENANT_ID }}
+          AZURE_AD_CLIENT_ID: ${{ secrets.AZURE_AD_CLIENT_ID }}
+          AZURE_AD_SECRET_VALUE: ${{ secrets.AZURE_AD_SECRET_VALUE }}
+        run: python -m unittest

README.md

@@ -309,6 +309,8 @@ docker stop -t 30 logzio-api-fetcher
 ```
 
 ## Changelog:
+- **0.3.1**:
+  - Handle internal logger configuration in memory instead of disk
 - **0.3.0**:
   - Add support for dockerhub audit logs
 - **0.2.2**:

e2e_tests/README.md

@@ -0,0 +1,66 @@
+# End-to-End Tests
+
+## Overview
+This guide provides instructions for developers on how to add new end-to-end (E2E) tests for different modules using the `ApiE2ETest` base class. It also outlines the necessary environment variables that need to be set.
+
+### Environment Variables
+
+Ensure the following environment variables are set:
+- `LOGZIO_API_TOKEN`: Your Logz.io API token.
+- `LOGZIO_SHIPPING_TOKEN`: Your Logz.io shipping token.
+
+Or any other environment variables that are required for the tests to run.
+
+### Adding New E2E Tests
+1. **Create a New Test File**: Add a new test file in the `e2e_tests/apis/` directory. Follow the naming convention `test_<module_name>_e2e.py`.
+2. **Extend the Base Class**: Inherit from the `ApiE2ETest` base class to leverage common setup and teardown functionalities.
+3. **Implement Module-Specific Methods**: Override the `module_specific_setup`, `module_specific_teardown`, and other necessary methods as needed.
+4. **Add Configuration Files**: Place any necessary configuration files in the `e2e_tests/testdata/` directory.
+
+### Utility Functions
+- `run_main_program`: Runs the main program with the specified configuration file and secrets map.
+- `search_logs`: Searches for logs in logzio based on the specified query.
+
+### Secrets Map
+The `secrets_map` is a dictionary that maps the secrets in the configuration file to the environment variables. This allows you to pass sensitive information such as tokens without hardcoding them in the configuration file.
+Example:
+```python
+secrets_map = {
+    "apis.0.dockerhub_token": "DOCKERHUB_TOKEN",
+    "apis.0.dockerhub_user": "DOCKERHUB_USER",
+    "logzio.token": "LOGZIO_SHIPPING_TOKEN",
+    "apis.0.additional_fields.type": "TEST_TYPE"
+}
+```
+### New E2E test example
+```python
+from e2e_tests.api_e2e_test import ApiE2ETest
+import unittest
+from os.path import abspath, dirname
+
+class TestNewModuleE2E(ApiE2ETest):
+
+    def module_specific_setup(self):
+        # Add module-specific setup here
+        pass
+
+    def module_specific_teardown(self):
+        # Add module-specific teardown here
+        pass
+
+    def test_new_module_e2e(self):
+        secrets_map = {
+            "apis.0.new_module_token": "NEW_MODULE_TOKEN",
+            "logzio.token": "LOGZIO_SHIPPING_TOKEN",
+            "apis.0.additional_fields.type": "TEST_TYPE"
+        }
+        curr_path = abspath(dirname(__file__))
+        config_path = f"{curr_path}/testdata/valid_new_module_config.yaml"
+        self.run_main_program(config_path=config_path, secrets_map=secrets_map)
+        logs = self.search_logs(f"type:{self.test_type}")
+        self.assertTrue(logs)
+        self.assertTrue(all([log.get("_source").get("eventType") == "new_event" for log in logs]))
+
+if __name__ == '__main__':
+    unittest.main()
+```

e2e_tests/api_e2e_test.py

@@ -0,0 +1,100 @@
+import os
+import threading
+import random
+import string
+import unittest
+import yaml
+
+from src.main import main
+from e2e_tests.utils.config_utils import update_config_tokens, delete_temp_files, validate_config_tokens
+from e2e_tests.utils.log_utils import search_data
+
+
+def print_yaml_content(file_path):
+    try:
+        with open(file_path, 'r') as file:
+            # Load the YAML content
+            yaml_content = yaml.safe_load(file)
+
+            # Print the YAML content
+            print(yaml.dump(yaml_content, sort_keys=False, default_flow_style=False))
+    except FileNotFoundError:
+        print(f"Error: The file {file_path} does not exist.")
+    except yaml.YAMLError as e:
+        print(f"Error parsing YAML file: {e}")
+
+
+class ApiE2ETest(unittest.TestCase):
+    """
+    Base class for API End-to-End tests. Provides common setup and teardown functionalities,
+    as well as utility methods for running the main program and searching logs in logzio.
+    """
+
+    def setUp(self):
+        """
+        Set up the test environment. Generates a random test type, sets environment variables,
+        and performs module-specific setup.
+        """
+        self.test_type = f"api-fetcher-e2e-test-{self.generate_random_string()}"
+        os.environ["TEST_TYPE"] = self.test_type
+        self.module_specific_setup()
+
+    def tearDown(self):
+        """
+        Tear down the test environment. Performs module-specific teardown and deletes temporary files.
+        """
+        self.module_specific_teardown()
+        delete_temp_files()
+
+    @staticmethod
+    def generate_random_string(length=5):
+        return ''.join(random.choices(string.ascii_lowercase + string.digits, k=length))
+
+    def set_configuration(self, config_path, secrets_map):
+        self.token_map = secrets_map
+        self.config_path = config_path
+        print(f"Configuration file: {self.config_path}")
+        print_yaml_content(self.config_path)
+        validate_config_tokens(self.token_map)
+        self.temp_config_path = update_config_tokens(self.config_path, self.token_map)
+        print(f"Temp configuration file: {self.config_path}")
+        print_yaml_content(self.temp_config_path)
+
+    def run_main_program(self, config_path, secrets_map, test=False):
+        """
+        Run the main program in a separate thread.
+
+        Args:
+            test (bool): Whether to run the program in test mode. Default is False.
+            config_path (str): The path to the configuration file.
+            secrets_map (dict): A dictionary mapping configuration tokens to environment variables.
+        """
+        self.set_configuration(config_path, secrets_map)
+        thread = threading.Thread(target=main, kwargs={"conf_path": self.temp_config_path, "test": test})
+        thread.daemon = True
+        thread.start()
+        thread.join(timeout=60)
+
+    def search_logs(self, query):
+        """
+        Search logs in logzio based on the provided query.
+
+        Args:
+            query (str): The query string to search for in the logs.
+
+        Returns:
+            list: A list of log entries that match the query.
+        """
+        return search_data(query)
+
+    def module_specific_setup(self):
+        """
+        Perform module-specific setup. This method can be overridden by subclasses.
+        """
+        pass
+
+    def module_specific_teardown(self):
+        """
+        Perform module-specific teardown. This method can be overridden by subclasses.
+        """
+        pass

e2e_tests/apis/test_e2e_azure.py

@@ -0,0 +1,35 @@
+import time
+
+from e2e_tests.api_e2e_test import ApiE2ETest
+import unittest
+from os.path import abspath, dirname
+
+
+class TestAzureE2E(ApiE2ETest):
+
+    def module_specific_setup(self):
+        # Add module-specific setup here if needed
+        pass
+
+    def module_specific_teardown(self):
+        # Add module-specific teardown here if needed
+        pass
+
+    def test_azure_data_in_logz(self):
+        secrets_map = {
+            "apis.0.azure_ad_tenant_id": "AZURE_AD_TENANT_ID",
+            "apis.0.azure_ad_client_id": "AZURE_AD_CLIENT_ID",
+            "apis.0.azure_ad_secret_value": "AZURE_AD_SECRET_VALUE",
+            "logzio.token": "LOGZIO_SHIPPING_TOKEN",
+            "apis.0.additional_fields.type": "TEST_TYPE"
+        }
+        curr_path = abspath(dirname(__file__))
+        config_path = f"{curr_path}/testdata/azure_api_conf.yaml"
+        self.run_main_program(config_path=config_path, secrets_map=secrets_map)
+        time.sleep(120)
+        logs = self.search_logs(f"type:{self.test_type}")
+        self.assertTrue(logs)
+
+
+if __name__ == '__main__':
+    unittest.main()

e2e_tests/apis/test_e2e_dockerhub.py

@@ -0,0 +1,52 @@
+import time
+import unittest
+from os.path import abspath, dirname
+from e2e_tests.api_e2e_test import ApiE2ETest
+import docker
+import requests
+from datetime import datetime, timedelta, timezone
+
+DOCKER_IMAGE = "logzio/dummy-docker-image"
+DOCKER_TAG = "latest"
+
+
+def was_tag_pushed_recently(image, tag, days=14):
+    url = f"https://hub.docker.com/v2/repositories/{image}/tags/{tag}"
+    response = requests.get(url)
+    if response.status_code == 200:
+        tag_info = response.json()
+        last_updated = datetime.strptime(tag_info['last_updated'], "%Y-%m-%dT%H:%M:%S.%fZ")
+        last_updated = last_updated.replace(tzinfo=timezone.utc)
+        return datetime.now(timezone.utc) - last_updated < timedelta(days=days)
+    return False
+
+
+class TestDockerhubE2E(ApiE2ETest):
+
+    def module_specific_setup(self):
+        curr_path = abspath(dirname(__file__))
+        self.client = docker.from_env()
+        self.dockerfile_path = f"{curr_path}/testdata/test.dockerfile"
+        if not was_tag_pushed_recently(DOCKER_IMAGE, DOCKER_TAG):
+            self.client.images.build(path=curr_path, dockerfile=self.dockerfile_path,
+                                     tag=f"{DOCKER_IMAGE}:{DOCKER_TAG}")
+            self.client.images.push(DOCKER_IMAGE, tag=DOCKER_TAG)
+
+    def test_dockerhub_audit_logs_e2e(self):
+        secrets_map = {
+            "apis.0.dockerhub_token": "DOCKERHUB_PASSWORD",
+            "apis.0.dockerhub_user": "DOCKERHUB_USER",
+            "logzio.token": "LOGZIO_SHIPPING_TOKEN",
+            "apis.0.additional_fields.type": "TEST_TYPE"
+        }
+        curr_path = abspath(dirname(__file__))
+        config_path = f"{curr_path}/testdata/valid_dockerhub_config.yaml"
+        self.run_main_program(config_path=config_path, secrets_map=secrets_map)
+        time.sleep(120)
+        logs = self.search_logs(f"type:{self.test_type}")
+        self.assertTrue(logs)
+        self.assertTrue(all([log.get("_source").get("eventType") == "auditevents" for log in logs]))
+
+
+if __name__ == '__main__':
+    unittest.main()

e2e_tests/apis/testdata/azure_api_conf.yaml

@@ -0,0 +1,16 @@
+apis:
+  - name: azure test
+    type: azure_graph
+    azure_ad_tenant_id: <<AZURE_AD_TENANT_ID>>
+    azure_ad_client_id: <<AZURE_AD_CLIENT_ID>>
+    azure_ad_secret_value: <<AZURE_AD_SECRET_VALUE>>
+    data_request:
+      url: https://graph.microsoft.com/v1.0/auditLogs/signIns
+    additional_fields:
+      type: azure_graph_shipping_test
+    days_back_fetch: 8
+    scrape_interval: 10
+
+logzio:
+  url: https://listener.logz.io:8071
+  token: SHipPIngtoKen

e2e_tests/apis/testdata/test.dockerfile

@@ -0,0 +1,2 @@
+FROM alpine:latest
+CMD ["echo", "Hello, World!"]