[WIP][REP][Serve] Add proposal for API allowing user-defined autoscaling and scheduling algorithms

[arcyleung]

This REP aims to extend the existing Ray serve.deployment functionality for users to define their own autoscaling and scheduling policy. The existing policies are request-queue-length scaling and power-of-2 scheduling, while the configs themselves can be modified (ie. target_ongoing_requests) the policy's algorithm and monitored heuristics currently cannot be changed by the user. For instance, if the user wants to specify scaling by number of SLA violations detected to proactively scale deployments, they cannot use a request queue length heuristic when each request has variable latency.

[arcyleung]

We ran some experiments comparing the behaviour of power of 2 scheduling/ max queue length autoscaling (Ray-like) with this adaptive SLA-aware scheduling/ autoscaling, under a scenario serving 2 LLM applications (wf1,wf2) simultaneously. An adaptive policy for both actions is beneficial for meeting SLAs beyond "lowest-latency" or "highest-throughput" goals, an get a compromise that is just right.