Helm Chart

[maheckathorn]

Hello,

Would a helm chart be a feature that the MISP project is interested in? If so, would this repo be the appropriate place for that feature or would a different or even new MISP Helm repo be more appropriate?

Helm does offer a good deployment method for running MISP on production workloads. The main project has some discussion about personal Helm charts, but nothing about an officially supported option.

[ostefano]

Yep definitely.

Let's start with a PR with this project and we can't take it from there.

[hodgigre]

Hello both. I'm just curious if you made any inroads on this front? On a related note I wonder whether these issue here has been resolved? Support: Split frontend/background workers · Issue #8616 · MISP/MISP (github.com) Thanks so much.

[ostefano]

Re first question you got to ask @maheckathorn , re second question you got to ask @iglocska

[maheckathorn]

We have working code that we are making use of internally. However, our process for open sourcing work is time consuming. I'm not going to be able to give a time line at this point.

[hodgigre]

Thank you for the update @maheckathorn. Are you using MISP on K8S in production? Has said architecture met your goals/objectives - any significant concerns or gaps? Would you recommend pursuit of this over native VM installs and related auto scaling technologies. We're actually targeting GKE. Thanks again.

[maheckathorn]

We're proving out making use of it in production. It's been working well. One of the benefits of using k8s over other scaling tech or native VM installs is access to the full k8s ecosystem. This allows us to use things like flux or ArgoCD for doing Continuous deployment via GitOPs, for one example. On cloud providers, we have also proved out swapping in database as a service offerings for the DB backend and that has been seamless.

[ostefano]

@maheckathorn this is becoming more and more desirable/critical.
Is there anything we can do on the MISP side to facilitate this?

[maheckathorn]

Not really, the ball is currently in our corner and we are beginning the legal conversations required to allow us to proceed to open sourcing our work.

[AxFrancois]

A helm chart would be very useful. I hope you can get over the administrative and legal burden soon👍

[jhuntwork]

Hello, I have also been looking at running this in a k8s cluster. One of the things I've noticed right away is that having multiple services (managed by supervisord) running inside one container is a little bit of an anti-pattern for k8s. And indeed, I've seen at times that supervisord fails to correctly re-launch a service like php-fpm. I have started refactoring things so that nginx, php-fpm and cron all run in independent containers with a shared volume for the codebase. This has resulted in much better stability, since k8s itself is able to manage the health/liveness of an individual process like nginx/php-fpm. However it does seem like some of fundamental designs of the app is the idea that an admin, through the web UI will manage supervisor workers. I'd like to gauge the interest in re-designing or adjusting these containers to be more k8s friendly?

[ostefano]

Hi @jhuntwork , in principle I have nothing against it as long as the changes are backward compatible with deployments where K8S is not used. Could you provide an MVP (maybe with just the cron service extracted) + helm chart?

What can not be changed though are the workers running via supervisor. If you have a proposal for an alternative you need to open an issue in the main MISP project.

[hodgigre]

Is the issue you're describing @jhuntwork not the same issue described here https://dev.to/pburgisser/misp-at-scale-on-kubernetes-2k31 and a proposed change here from a couple years back for which I can't understand the status of the merge. MISP/MISP#8616
MISP/MISP#8651

And/or this. coolacid/docker-misp#184

[jhuntwork]

Hi @jhuntwork , in principle I have nothing against it as long as the changes are backward compatible with deployments where K8S is not used. Could you provide an MVP (maybe with just the cron service extracted) + helm chart?

What can not be changed though are the workers running via supervisor. If you have a proposal for an alternative you need to open an issue in the main MISP project.

@ostefano Nice, thank you for the feedback. I made a fork and started on some changes, here: master...jhuntwork:misp-docker:master - What is there is mostly working for me, although I have one or two small changes I want to add. I will turn that into a pull request soon so that there is a better place to discuss specific changes. Mostly these changes move initial configuration logic away from service starting and management logic, which is where things would diverge a bit for k8s.

[jhuntwork]

Is the issue you're describing @jhuntwork not the same issue described here https://dev.to/pburgisser/misp-at-scale-on-kubernetes-2k31 and a proposed change here from a couple years back for which I can't understand the status of the merge. MISP/MISP#8616 MISP/MISP#8651

And/or this. coolacid/docker-misp#184

@hodgigre Thank you for the links. Yes that first document does seem to well describe the issue, although I agree it's hard to assess the current state of things. I will be needing to run and support running MISP on k8s (EKS) likely for some time, so I'll be happy to use whatever work has already been done and published as well as contribute to any effort there.