Extension - Monitor CPU and bail if approaching 80% utilization to avoid throttling

[huntharo]

Motivations

• Lambda has a model of sending no more than 1 request to an exec env every 100 ms
• When allocating, say, 1/8th of a CPU, you are really saying you want 12.5 ms of CPU in a 100 ms time slice - if you go over this you get throttled (just like with containers)
• The dispatch model of only 1 request per 100 ms prevents problems with the 1st request causing a 2nd request on the same exec env to be throttled
• With lambda-dispatch this model is broken because we allow concurrent requests, which makes throttling more likely, and the consequences are devastating to throughput and predictable response times

Solution Pattern

• This problem can be mitigated by monitoring CPU usage vs CPU request/limit and telling the router we need to go away when we approach a threshold (e.g. 70% or 80%) of usage of the CPU amount per 100 ms timeslice starting with 0 ms as the invocation time
• When the lambda exec env returns, it won't be invoked until the 100 ms timeslice completes, so the requests will get routed to an exec env that is not likely to be throttled
• Caveat: this cannot prevent all throttling, but it can avoid repeatedly encountering it in a single instance and it avoids a throttled instance being sent many additional requests that would exacerbate the problem

Acceptance Criteria

• TBC