Integration tests for kfp-profile-controller not working for track/2.0-alpha.7

[misohu]

Bug Description

Integration test for kfp-profile-controller are not working for track/2.0-alpha.7. After executing the tox environment for integration tests they fail while trying to deploy built kfp-api charm to juju with message.

To Reproduce

1. clone repo
2. git checkout track/2.0-alpha.7
3. cd charms/kfp-profile-controller
4. tox -e integration -- --model kubeflow --keep-models -vv -s

Environment

juju - 2.9.47-ubuntu-amd64
microk8s - v1.24.17

Relevant Log Output

============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.4.2, pluggy-1.3.0 -- /home/runner/work/kfp-operators/kfp-operators/.tox/integration/bin/python
cachedir: /home/runner/work/kfp-operators/kfp-operators/.tox/integration/.pytest_cache
rootdir: /home/runner/work/kfp-operators/kfp-operators/charms/kfp-profile-controller
configfile: pyproject.toml
plugins: operator-0.29.0, asyncio-0.21.1, anyio-4.0.0
asyncio: mode=auto
collecting ... collected 4 items

tests/integration/test_charm.py::test_build_and_deploy 
-------------------------------- live log setup --------------------------------
INFO     pytest_operator.plugin:plugin.py:647 Adding model github-pr-7e3ce-microk8s:test-charm-leud on cloud microk8s
-------------------------------- live log call ---------------------------------
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/integration/tmp/pytest/test-charm-leud0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-profile-controller
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-profile-controller in 126.22s
INFO     test_charm:test_charm.py:29 Built charm /home/runner/work/kfp-operators/kfp-operators/.tox/integration/tmp/pytest/test-charm-leud0/charms/kfp-profile-controller_ubuntu-20.04-amd64.charm
INFO     juju.model:model.py:1984 Deploying local:kubernetes/kfp-profile-controller-0
INFO     juju.model:model.py:1984 Deploying ch:amd64/focal/minio-277
INFO     juju.model:model.py:1984 Deploying ch:amd64/focal/kubeflow-profiles-362
INFO     juju.model:model.py:1984 Deploying ch:amd64/focal/metacontroller-operator-250
INFO     juju.model:model.py:2621 Waiting for model:
  kfp-profile-controller/0 [executing] waiting: List of <ops.model.Relation object-storage:0> versions not found for apps: minio
  minio/0 [executing] maintenance: installing charm software
  kubeflow-profiles/0 [allocating] waiting: installing agent
  metacontroller-operator/0 [allocating] waiting: installing agent
FAILED
tests/integration/test_charm.py::test_profile_and_resources_creation 
-------------------------------- live log setup --------------------------------
INFO     httpx:_client.py:1013 HTTP Request: PATCH https://10.1.0.26:16443/apis/kubeflow.org/v1/profiles/profilename?fieldManager=kfp-profile-controller "HTTP/1.1 201 Created"
XFAIL (aborted)
tests/integration/test_charm.py::test_model_resources XFAIL (aborted)
tests/integration/test_charm.py::test_minio_config_changed XFAIL
------------------------------ live log teardown -------------------------------
INFO     pytest_operator.plugin:plugin.py:783 Model status:

Model            Controller                Cloud/Region        Version  SLA          Timestamp
test-charm-leud  github-pr-7e3ce-microk8s  microk8s/localhost  2.9.48   unsupported  08:20:36Z

App                      Version                Status   Scale  Charm                    Channel        Rev  Address         Exposed  Message
kfp-profile-controller   python:3.7             error        1  kfp-profile-controller                    0                  no       creating or updating custom resources: getting custom resources: attempt count exceeded: getting custom resource defi...
kubeflow-profiles                               waiting      1  kubeflow-profiles        latest/edge    362  10.152.183.159  no       installing agent
metacontroller-operator                         waiting      1  metacontroller-operator  latest/edge    250  10.152.183.150  no       installing agent
minio                    res:oci-image@1755999  waiting      1  minio                    latest/stable  277  10.152.183.229  no       

Unit                        Workload     Agent      Address      Ports              Message
kfp-profile-controller/0*   waiting      idle                                       waiting for container
kubeflow-profiles/0*        maintenance  executing  10.1.43.203                     (config-changed) Waiting for pod startup to complete
metacontroller-operator/0*  maintenance  executing  10.1.43.205                     (install) installing charm software
minio/0*                    waiting      executing  10.1.43.206  9000/TCP,9001/TCP  waiting for container



INFO     pytest_operator.plugin:plugin.py:789 Juju error logs:


INFO     pytest_operator.plugin:plugin.py:805 juju-crashdump finished [0]
INFO     pytest_operator.plugin:plugin.py:877 Resetting model test-charm-leud...
INFO     pytest_operator.plugin:plugin.py:866    Destroying applications kfp-profile-controller
INFO     pytest_operator.plugin:plugin.py:866    Destroying applications minio
INFO     pytest_operator.plugin:plugin.py:866    Destroying applications kubeflow-profiles
INFO     pytest_operator.plugin:plugin.py:866    Destroying applications metacontroller-operator
INFO     pytest_operator.plugin:plugin.py:882 Not waiting on reset to complete.
INFO     pytest_operator.plugin:plugin.py:855 Forgetting main...
INFO     httpx:_client.py:1013 HTTP Request: DELETE https://10.1.0.26:16443/apis/kubeflow.org/v1/profiles/profilename "HTTP/1.1 200 OK"


=================================== FAILURES ===================================
____________________________ test_build_and_deploy _____________________________
Traceback (most recent call last):
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
    result: Optional[TResult] = func()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
    raise e
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
    item.runtest()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 1792, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 493, in __call__
    return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_manager.py", line 115, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 152, in _multicall
    return outcome.get_result()
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_result.py", line 114, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pluggy/_callers.py", line 77, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 532, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/runner/work/kfp-operators/kfp-operators/charms/kfp-profile-controller/tests/integration/test_charm.py", line 62, in test_build_and_deploy
    await ops_test.model.wait_for_idle(status="active", timeout=60 * 10)
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2613, in wait_for_idle
    _raise_for_status(errors, "error")
  File "/home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/juju/model.py", line 2525, in _raise_for_status
    raise error_type("{}{} in {}: {}".format(
juju.errors.JujuAppError: App in error: kfp-profile-controller
------------------------------ Captured log setup ------------------------------
INFO     pytest_operator.plugin:plugin.py:647 Adding model github-pr-7e3ce-microk8s:test-charm-leud on cloud microk8s
------------------------------ Captured log call -------------------------------
INFO     pytest_operator.plugin:plugin.py:526 Using tmp_path: /home/runner/work/kfp-operators/kfp-operators/.tox/integration/tmp/pytest/test-charm-leud0
INFO     pytest_operator.plugin:plugin.py:975 Building charm kfp-profile-controller
INFO     pytest_operator.plugin:plugin.py:980 Built charm kfp-profile-controller in 126.22s
INFO     test_charm:test_charm.py:29 Built charm /home/runner/work/kfp-operators/kfp-operators/.tox/integration/tmp/pytest/test-charm-leud0/charms/kfp-profile-controller_ubuntu-20.04-amd64.charm
INFO     juju.model:model.py:1984 Deploying local:kubernetes/kfp-profile-controller-0
INFO     juju.model:model.py:1984 Deploying ch:amd64/focal/minio-277
INFO     juju.model:model.py:1984 Deploying ch:amd64/focal/kubeflow-profiles-362
INFO     juju.model:model.py:1984 Deploying ch:amd64/focal/metacontroller-operator-250
INFO     juju.model:model.py:2621 Waiting for model:
  kfp-profile-controller/0 [executing] waiting: List of <ops.model.Relation object-storage:0> versions not found for apps: minio
  minio/0 [executing] maintenance: installing charm software
  kubeflow-profiles/0 [allocating] waiting: installing agent
  metacontroller-operator/0 [allocating] waiting: installing agent
=============================== warnings summary ===============================
tests/integration/test_charm.py::test_build_and_deploy
tests/integration/test_charm.py::test_build_and_deploy
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:218: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    self._drain_lock = asyncio.Lock(loop=loop)

tests/integration/test_charm.py: 18 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:977: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.sleep(self.ping_interval, loop=self.loop)

tests/integration/test_charm.py: 341 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:911: DeprecationWarning: 'with (yield from lock)' is deprecated use 'async with lock' instead
    with (yield from self._drain_lock):

tests/integration/test_charm.py: 323 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:416: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait(

tests/integration/test_charm.py: 16 warnings
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:988: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::test_minio_config_changed
tests/integration/test_charm.py::test_minio_config_changed
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:532: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::test_minio_config_changed
tests/integration/test_charm.py::test_minio_config_changed
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:554: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

tests/integration/test_charm.py::test_minio_config_changed
tests/integration/test_charm.py::test_minio_config_changed
  /home/runner/work/kfp-operators/kfp-operators/.tox/integration/lib/python3.8/site-packages/websockets/protocol.py:1077: DeprecationWarning: The loop argument is deprecated since Python 3.8, and scheduled for removal in Python 3.10.
    yield from asyncio.wait_for(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/integration/test_charm.py::test_build_and_deploy - juju.errors.JujuAppError: App in error: kfp-profile-controller
============ 1 failed, 3 xfailed, 706 warnings in 170.01s (0:02:50) ============
integration: exit 1 (171.13 seconds) /home/runner/work/kfp-operators/kfp-operators/charms/kfp-profile-controller> pytest -vv --tb native --asyncio-mode=auto /home/runner/work/kfp-operators/kfp-operators/charms/kfp-profile-controller/tests/integration --log-cli-level=INFO -s pid=14702
  integration: FAIL code 1 (193.29=setup[22.16]+cmd[171.13] seconds)
  evaluation failed :( (193.37 seconds)
kfp-profile-controller-integration: exit 1 (193.55 seconds) /home/runner/work/kfp-operators/kfp-operators> tox -c charms/kfp-profile-controller -e integration -- pid=14071
  kfp-profile-controller-integration: FAIL code 1 (194.81=setup[1.26]+cmd[193.55] seconds)
  evaluation failed :( (194.92 seconds)
Error: Process completed with exit code 1.

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5521.

This message was autogenerated

[misohu]

I found out that there are wrong versions for kubeflow-profiles and metacontroller-operator so I pinned those to versions from kubeflow bundle 1.7. I also had to add raise_on_error=False in the waiting function for kubeflow-profiles-controller as the deployment was sometimes going to the error state (This edge case might have been missed in previous PRs as it happend occasionally, moreover other charm's integration tests have the raise_on_error=False set).