feat: reduce image size further

[dhaustein]

Reduces the size of the resulting image to around 400 MB.

• Uses multi-stage build process.
• Python dependencies are packaged into one venv.
• Java JRE is created custom, only keeping the modules Drools needs.
• Both the venv and the JRE are copied to the second stage of the build based on stream9-minimal image to reduce the size even further.

Comparison with the original image:

$  buildah build --no-cache --tag localhost/ansible-rulebook:slim -f Dockerfile

$  podman images --format "table {{.Repository}}:{{.Tag}} {{.Size}}" | rg ansible-rulebook
31128877f949  localhost/ansible-rulebook:slim                  323 MB
5cbebc6a4a9b  localhost/ansible-rulebook:latest                702 MB

Note: When built with podman, the image size will be ~100 MB larger for some reason (despite buildah being used internally in podman-build), even with compression forced on.

[dhaustein]

TODO

• fix E2E tests failing on CI https://github.com/ansible/ansible-rulebook/actions/runs/7411894851/job/20167469002?pr=639
• use regular stream9 image instead of -development in the build stage feat: reduce image size further #639 (comment)
• add comments

[dhaustein]

@Alex-Izquierdo I remember you wanted to use python dependencies defined in the requirements file(s) instead of hard-coding them in the Dockerfile. I will want to move onto that in a future PR. Didn't want to mix that into this already big change.

[mkanoor]

@dhaustein I think we should also ensure that all the source plugins that we ship in ansible.eda collection work with the slim image and all dependencies are satisfied.

[dhaustein]

@dhaustein I think we should also ensure that all the source plugins that we ship in ansible.eda collection work with the slim image and all dependencies are satisfied.

@mkanoor I don't understand what you mean 100% but do you simply mean that I should test the new image with the EDA test suite from https://github.com/ansible/eda-qa/ ?

[mkanoor]

@dhaustein I think we should also ensure that all the source plugins that we ship in ansible.eda collection work with the slim image and all dependencies are satisfied.

@mkanoor I don't understand what you mean 100% but do you simply mean that I should test the new image with the EDA test suite from https://github.com/ansible/eda-qa/ ?

@dhaustein @ttuffin Currently there are 15 source plugins that we support in ansible.eda collection, I am not sure if QE has tests that cover each and every source plugin that we have. Each of these plugins have dependencies on other python packages, which is defined in the collection requirements. Our DE that we built should support these 15 source plugins, if there is a package that is missing then the rulebook using our official ansible.eda collection source will fail. At the least I am thinking that we should have some basic tests for each of these source plugins. To test the sources we would need access to other cloud providers and I am not sure if that integration test is happening.

[dhaustein]

@mkanoor Testing it with every plugin is a rather tall order and I am not sure the EDA test suite covers everything. But I will see what I can do.

[dhaustein]

@ansible/eda-maintainers Following up on the discussion from #638 what is the purpose of the image?

Is it a runtime image that provides decision environment + CLI or is it a dev image that others can extend and built upon? I don't think I can make it both while maintaining small size.

[ttuffin]

@mkanoor Testing it with every plugin is a rather tall order and I am not sure the EDA test suite covers everything. But I will see what I can do.

@mkanoor @dhaustein we have integration tests in the collection repository for this purpose, and a workflow that installs the dependencies and runs the tests. That said, we only have coverage for four of the source plugins so we definitely need to look at increasing the coverage there.

[ttuffin]

@dhaustein first and foremost it is a base DE + CLI, and since it's used in our CI pipelines we want to keep it as small as possible. Allowing for extensibility comes second to that, imo.

[Alex-Izquierdo]

Hi @dhaustein the image is the base DE, which implicitly serves as an execution environment for the CLI. This makes it ideal for CI use, particularly for the eda-server. It's not intended for development purposes but rather for general usage. I believe that focusing on size, a reasonable requirement for a production-ready image, doesn't conflict with the goal of extensibility. Users can always build their own images if they have more complex or specific needs.

Regarding the tests for our plugins in the collection I don't think that the API test suite is the right place for it. All the plugins there should have integration tests with ansible-rulebook and the source repo is probably the ideal place for it, eventually we could use them to test this image as @mkanoor suggested. I don't think he is proposing a blocker for this PR, I think he just wants to make sure that the collection integrates well with the image or the CLI. We could just do a manual check for now, but I think it is an interesting milestone for later iterations. I believe we already have a ticket in our internal Jira for it.