Allow optional configurable limits and requests on CPU and RAM for containers

[seanhamlin]

Related to #2957

At present nginx/php are spawned with incredibly tiny requests for both the Nginx and PHP containers:

          resources:
            requests:
              cpu: 10m
              memory: 10Mi

This is also hardcoded.

Impact

If you are running a highly authenticated application on a small cluster, it is hard to get the EKS cluster to scale horizontally. The HPA will load more pods on to the already CPU exhausted nodes.

Ideal solution

We are able to define the requests per container, ideally, there would be a way to configure PHP to have:

          resources:
            requests:
              cpu: 2
              memory: 4Gi

This way, as the HPA scaled in more pods, they would force the cluster to scale horizontally in line with the requests.

Potentially more labels in docker-compose.yml?

  php:
    build:
      context: .
      dockerfile: php.dockerfile
    labels:
      lagoon.type: nginx-php-persistent
      lagoon.name: nginx
      lagoon.persistent: /app/web/sites/default/files/
	  lagoon.requests.cpu: 2
      lagoon.requests.memory: 8Gi

[smlx]

I don't think this should be user configurable (too easy to break things) but potentially having some way for an administrator to override the default requests per project/environment would be helpful.

[seanhamlin]

I think that would be perfect.

[seanhamlin]

This would have come in handy with a load test for an application that was heavy on CPU, and the cluster was only moderately sized (5 nodes).