Fix and restructure functional tests

tests/functional/README.md

@@ -3,41 +3,42 @@ There are 2 main types of functional tests for the Hardware Observer charm - tho
 
 Here, "real hardware" refers to machines that are not VMs or containers and have access to real hardware resources like RAID cards and BMC management tools.
 
+Note: the built charm must be present in the root of the project's directory for the tests to run.
+
 ## Hardware Independent Tests
 These are the tests for hardware observer that do not require any real hardware.
 
-Hardware independent tests are run on every PR / weekly scheduled test run. They belong to the `TestCharm` class in the `test_charm.py` module.
+Hardware independent tests are run on every PR / weekly scheduled test run.
 
 These include:
 * Testing whether juju config changes produce the required results
-* Check whether the exporter systemd service starts and stops correctly
-* Test exporter is stopped and related files removed on removal of charm
-
-and more.
 
-Running these tests is as simple as executing the `make functional` command.
+Running these tests is as simple as executing the `tox -e func -- -v`
 
 ## Hardware Dependent Tests
 These are the tests that depend on real hardware to be executed. This is performed manually when required, for example - validating the charm's full functionality before a new release.
 
 Hardware dependent tests are present in the `TestCharmWithHW` class in the `test_charm.py` module. The pytest marker `realhw` has been added to this class (which would include all the tests in this class).
 
-These tests will only be executed if the `--collectors` option for pytest is provided some value. Otherwise, all these tests are skipped (this is done by checking for the presence of the `realhw` marker mentioned earlier.)
+These tests will only be executed if the `--realhw` option for pytest is provided. Additionally, the `--collectors` option with space separated values can be provided, if specific hardware is present. Check the `conftest.py` for options. Otherwise, all these tests are skipped (this is done by checking for the presence of the `realhw` marker mentioned earlier.)
 
-Note: The `test_build_and_deploy` function sets up the test environment for both types of tests.
+Note: The operator must set up a test model with the machine added beforehand. The machine must be an actual host, containers or VMs won't work.
 
 Some of these tests include:
 * Check if all collectors are detected in the exporter config file
 * Test if metrics are available at the expected endpoint
 * Test if metrics specific to the collectors being tested are available
+* Test if smarctl-exporter snap is installed and running
+* Test if Nvidia drivers and dcgm-exporter snap are installed
 
 and more.
 
 In order to run these tests, a couple of prerequisite steps need to be completed.
 1. Setup test environment
 2. Add environment variables for Redfish credentials.
 3. Setup required resource files
-4. Find supported collectors
+4. Determine if the machine has Nvidia GPUs and add the `--nvidia` flag is present.
+5. Find supported collectors
 
 ### 1. Setup test environment
 For the hardware dependent tests, we add the test machine beforehand and the bundle only handles deploying the applications to this machine.
@@ -82,7 +83,7 @@ Note: The tests expect these resources to be named exactly in the manner provide
 ### 4. Find supported collectors
 Note down all the collectors supported by the machine as they need to be provided to pytest as part of its CLI arguments.
 
-This is done by passing the required collectors in a space-separated manner via the `FUNC_ARGS` environment variable to the make target.
+This is done by passing the required collectors in a space-separated manner via `--collector` option to the tox target.
 
 The supported collectors can be found by checking the output of the `lshw` command (for RAID cards) or checking availability of Redfish and IPMI on the BMC.
 
@@ -92,7 +93,7 @@ The supported collectors can be found by checking the output of the `lshw` comma
 
 After ensuring the prerequisite steps are complete, the final command to run the tests would look something like this:
 ```
-FUNC_ARGS="--model test --collectors ipmi_dcmi ipmi_sel ipmi_sensor redfish mega_raid" make functional
+tox -e func -- -v --realhw --model test --collectors ipmi_dcmi ipmi_sel ipmi_sensor redfish mega_raid --keep-models
 ```
 
 This would pass the required collectors to tox which then sends it to the pytest command and starts the hardware dependent tests.

tests/functional/conftest.py

@@ -1,5 +1,6 @@
 import logging
 import platform
+from pathlib import Path
 
 import pytest
 from utils import RESOURCES_DIR, Resource
@@ -18,6 +19,18 @@ def pytest_addoption(parser):
         help="Set base for the applications.",
     )
 
+    parser.addoption(
+        "--realhw",
+        action="store_true",
+        help="Enable real hardware testing.",
+    )
+
+    parser.addoption(
+        "--nvidia",
+        action="store_true",
+        help="Enable NVIDIA GPU support for testing with real hardware.",
+    )
+
     parser.addoption(
         "--collectors",
         nargs="+",
@@ -42,6 +55,16 @@ def base(request):
     return request.config.getoption("--base")
 
 
+@pytest.fixture(scope="module")
+def nvidia_present(request):
+    return request.config.getoption("--nvidia")
+
+
+@pytest.fixture(scope="module")
+def realhw(request):
+    return request.config.getoption("--realhw")
+
+
 @pytest.fixture(scope="module")
 def architecture():
     machine = platform.machine()
@@ -60,20 +83,7 @@ def pytest_configure(config):
 
 
 def pytest_collection_modifyitems(config, items):
-    if config.getoption("collectors"):
-        # --collectors provided, skip hw independent tests
-        skip_hw_independent = pytest.mark.skip(
-            reason="Hardware independent tests are skipped since --collectors was provided."
-        )
-        for item in items:
-            # skip TestCharm tests where "realhw" marker is not present
-            # we don't want to skip test_setup_and_build, test_required_resources,
-            # test_cos_agent_relation and test_redfish_credential_validation
-            # even for hw independent tests
-            # so we also check for the abort_on_fail marker
-            if "realhw" not in item.keywords and "abort_on_fail" not in item.keywords:
-                item.add_marker(skip_hw_independent)
-    else:
+    if not config.getoption("--realhw"):
         # skip hw dependent tests in TestCharmWithHW marked with "realhw"
         skip_hw_dependent = pytest.mark.skip(
             reason="Hardware dependent test. Provide collectors with the --collectors option."
@@ -100,31 +110,31 @@ def resources() -> list[Resource]:
         Resource(
             resource_name=TPR_RESOURCES.get(HWTool.STORCLI),
             file_name="storcli.deb",
-            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.STORCLI)[0].replace(
+            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.STORCLI).replace(
                 "collector.", ""
             ),
             bin_name=HWTool.STORCLI.value,
         ),
         Resource(
             resource_name=TPR_RESOURCES.get(HWTool.PERCCLI),
             file_name="perccli.deb",
-            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.PERCCLI)[0].replace(
+            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.PERCCLI).replace(
                 "collector.", ""
             ),
             bin_name=HWTool.PERCCLI.value,
         ),
         Resource(
             resource_name=TPR_RESOURCES.get(HWTool.SAS2IRCU),
             file_name="sas2ircu",
-            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.SAS2IRCU)[0].replace(
+            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.SAS2IRCU).replace(
                 "collector.", ""
             ),
             bin_name=HWTool.SAS2IRCU.value,
         ),
         Resource(
             resource_name=TPR_RESOURCES.get(HWTool.SAS3IRCU),
             file_name="sas3ircu",
-            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.SAS3IRCU)[0].replace(
+            collector_name=HARDWARE_EXPORTER_COLLECTOR_MAPPING.get(HWTool.SAS3IRCU).replace(
                 "collector.", ""
             ),
             bin_name=HWTool.SAS3IRCU.value,
@@ -146,3 +156,23 @@ def required_resources(resources: list[Resource], provided_collectors: set) -> l
             required_resources.append(resource)
 
     return required_resources
+
+
+@pytest.fixture()
+def charm_path(base: str, architecture: str) -> Path:
+    """Fixture to determine the charm path based on the base and architecture."""
+    glob_path = f"hardware-observer_*{base.replace('@', '-')}-{architecture}*.charm"
+    paths = list(Path(".").glob(glob_path))
+
+    if not paths:
+        raise FileNotFoundError(f"The path for the charm for {base}-{architecture} is not found.")
+
+    if len(paths) > 1:
+        raise FileNotFoundError(
+            f"Multiple charms found for {base}-{architecture}. Please provide only one."
+        )
+
+    # The bundle will need the full path to the charm
+    path = paths[0].absolute()
+    log.info(f"Using charm path: {path}")
+    return path