diff --git a/examples/webserver/concurrency-limits/README.md b/examples/webserver/concurrency-limits/README.md new file mode 100644 index 00000000..14aeab27 --- /dev/null +++ b/examples/webserver/concurrency-limits/README.md @@ -0,0 +1,139 @@ +# Helidon SE Concurrency Limits Example + +This example demonstrates the Concurrency Limits feature of Helidon WebServer. +For other approaches for rate limiting see the [Rate Limit Example](../ratelimit). + +The Concurrency Limits feature provides mechanisms to limit concurrent execution of incoming requests. +Currently two algorithms are supported: + +1. Fixed: A semaphore based limit that supports queuing for a permit and timeout on the queue. +2. AIMD: Additive-increase/multiplicative-decrease. Uses a feedback control algorithm that combines linear growth of the request limit when there is no congestion with an exponential reduction when congestion is detected. + +The example does the following: + +1. Defines two ports for accepting incoming requests: 8080 and 8088 +2. 8080 is configured to use the Fixed concurrency limit algorithm +3. 8088 is configured to use the AIMD concurrency limit algorithm + +The server has two endpoints: `fixed/sleep` and `aimd/sleep`. Both sleep for the specified number of seconds to simulate a slow workload. + +These values are configured in [`application.yaml`](./src/main/resources/application.yaml) + +## Build and run + +Build and start the server: +```shell +mvn package +java -jar target/helidon-examples-webserver-concurrencylimits.jar +``` + +## Exercise the application + +### Fixed Concurrency Limit + +The server is configured to process 6 requests concurrently with a backlog queue of depth 4. Therefore it can handle bursts of up to 10 requests at a time. + +Send a burst of 15 concurrent requests to the `fixed/sleep` endpoint (each requesting a sleep of 3 seconds): +```shell +curl -s \ + --noproxy '*' \ + -o /dev/null \ + --parallel \ + --parallel-immediate \ + -w "%{http_code}\n" \ + "http://localhost:8080/fixed/sleep/3?c=[1-15]" +``` + +When the 15 concurrent requests hit the server: + +* 5 of the requests are rejected with a 503 because they exceed the size of the number of concurrent requests limit (6) plus the size of the queue (4). +* 10 of the requests are accepted by the server +* The first 6 of those return a 200 after sleeping for 3 seconds +* The next 4 requests are then processed from the queue and return a 200 after sleeping for 3 more seconds. + +You will see this in the output: +``` +503 +503 +503 +503 +503 +# Three second pause +200 +200 +200 +200 +200 +200 +# Three second pause +200 +200 +200 +200 +``` + +### AIMD Concurrency Limit + +The AIMD limiter is [adaptive](https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease) +and will adjust according to the workload and responsiveness of the server. +If the server successfully processes requests then the limiter will increase limits (up to a max limit). +If the server starts to fail or timeout requests then the limiter will reduce limits (down to a min limit). + +In this example we set the initial and minimum limit to 6 and a max limit of 10 concurrent requests. +We configure the timeout as 3 seconds. So requests that can't be serviced within 3 seconds fail -- which can lead to a reduction of the current limit. + +First execute 15 concurrent requests, each sleeping for 3 seconds. + +```shell +curl -s \ + --noproxy '*' \ + -o /dev/null \ + --parallel \ + --parallel-immediate \ + -w "%{http_code}\n" \ + "http://localhost:8088/aimd/sleep/3?c=[1-15]" +``` + +This will process 6 request (the currently configured limit), and fail 9: + +``` +503 +503 +503 +503 +503 +503 +503 +503 +503 +# Pause for 3 seconds +200 +200 +200 +200 +200 +200 +``` + +Repeat the curl command and you will see the same results. Because we have the AIMD timeout set to 3 seconds, and our sleep operation is taking 3 seconds, the current limit is never increased. + +Now repeat the curl command, but specify a sleep time of 2 seconds: + +```shell +curl -s \ + --noproxy '*' \ + -o /dev/null \ + --parallel \ + --parallel-immediate \ + -w "%{http_code}\n" \ + "http://localhost:8088/aimd/sleep/2?c=[1-15]" +``` + +As before the first invocation will process 6 requests. But as you +repeat the curl command you will see more requests are +successful until you hit the max concurrency limit of 10. Since +each request is now taking 2 seconds (and not 3), the limiter +is able to adapt and increase the current limit up to the maximum. + +Now go back and run the curl command a few times using a sleep time +of 3, and you will see the limiter adapt again and lower the current limit. diff --git a/examples/webserver/concurrency-limits/pom.xml b/examples/webserver/concurrency-limits/pom.xml new file mode 100644 index 00000000..462f246d --- /dev/null +++ b/examples/webserver/concurrency-limits/pom.xml @@ -0,0 +1,86 @@ + + + + 4.0.0 + + io.helidon.applications + helidon-se + 4.2.0-SNAPSHOT + + + io.helidon.examples.webserver + helidon-examples-webserver-concurrencylimits + 1.0.0-SNAPSHOT + Helidon Examples WebServer Concurrency Limits + + + io.helidon.examples.webserver.concurrencylimits.Main + + + + + io.helidon.webserver + helidon-webserver + + + io.helidon.config + helidon-config-yaml + + + io.helidon.fault-tolerance + helidon-fault-tolerance + + + io.helidon.logging + helidon-logging-jul + runtime + + + org.junit.jupiter + junit-jupiter-api + test + + + org.hamcrest + hamcrest-all + test + + + io.helidon.webserver.testing.junit5 + helidon-webserver-testing-junit5 + test + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + copy-libs + + + + + + diff --git a/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/Main.java b/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/Main.java new file mode 100644 index 00000000..5c06bc01 --- /dev/null +++ b/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/Main.java @@ -0,0 +1,75 @@ +/* + * Copyright (c) 2024 Oracle and/or its affiliates. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.helidon.examples.webserver.concurrencylimits; + +import io.helidon.config.Config; +import io.helidon.logging.common.LogConfig; +import io.helidon.webserver.WebServer; +import io.helidon.webserver.WebServerConfig; +import io.helidon.webserver.http.HttpRouting; + +/** + * The application main class. + */ +public class Main { + + private static final System.Logger LOGGER = System.getLogger(Main.class.getName()); + + /** + * Cannot be instantiated. + */ + private Main() { + } + + /** + * Application main entry point. + * @param args command line arguments. + */ + public static void main(String[] args) { + + // load logging configuration + LogConfig.configureRuntime(); + + // initialize global config from default configuration + Config config = Config.create(); + Config.global(config); + + // Default port uses the fixed limiter + // "aimd" port uses the AIMD limiter + WebServerConfig webserverConfig = WebServer.builder() + .config(config.get("server")) + .routing(Main::fixedRouting) + .routing("aimd", Main::aimdRouting) + .buildPrototype(); + + WebServer webserver = webserverConfig.build().start(); + + LOGGER.log(System.Logger.Level.INFO, "WEB server is up! http://localhost:" + webserver.port() + "/fixed/sleep" + + " " + "http://localhost:" + webserver.port("aimd") + "/aimd/sleep"); + } + + /** + * Updates HTTP Routing. + */ + static void fixedRouting(HttpRouting.Builder routing) { + routing.register("/fixed", new SleepService()); + } + + static void aimdRouting(HttpRouting.Builder routing) { + routing.register("/aimd", new SleepService()); + } +} diff --git a/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/SleepService.java b/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/SleepService.java new file mode 100644 index 00000000..f8718457 --- /dev/null +++ b/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/SleepService.java @@ -0,0 +1,63 @@ +/* + * Copyright (c) 2024 Oracle and/or its affiliates. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.helidon.examples.webserver.concurrencylimits; + +import io.helidon.webserver.http.HttpRules; +import io.helidon.webserver.http.HttpService; +import io.helidon.webserver.http.ServerRequest; +import io.helidon.webserver.http.ServerResponse; + +class SleepService implements HttpService { + + private static final System.Logger LOGGER = System.getLogger(SleepService.class.getName()); + + SleepService() { + } + + @Override + public void routing(HttpRules rules) { + rules.get("/sleep/{seconds}", this::sleepHandler); + } + + /** + * Sleep for a specified number of seconds. + * The optional path parameter controls the number of seconds to sleep. Defaults to 1 + * + * @param request server request + * @param response server response + */ + private void sleepHandler(ServerRequest request, ServerResponse response) { + int seconds = request.path().pathParameters().first("seconds").asInt().orElse(1); + response.send(String.valueOf(sleep(seconds))); + } + + /** + * Sleep current thread. + * + * @param seconds number of seconds to sleep + * @return number of seconds requested to sleep + */ + private static int sleep(int seconds) { + LOGGER.log(System.Logger.Level.DEBUG, Thread.currentThread() + ": Sleeping for " + seconds + " seconds"); + try { + Thread.sleep(seconds * 1_000L); + } catch (InterruptedException e) { + LOGGER.log(System.Logger.Level.WARNING, e); + } + return seconds; + } +} diff --git a/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/package-info.java b/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/package-info.java new file mode 100644 index 00000000..2cd5203c --- /dev/null +++ b/examples/webserver/concurrency-limits/src/main/java/io/helidon/examples/webserver/concurrencylimits/package-info.java @@ -0,0 +1,17 @@ +/* + * Copyright (c) 2024 Oracle and/or its affiliates. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.helidon.examples.webserver.concurrencylimits; diff --git a/examples/webserver/concurrency-limits/src/main/resources/application.yaml b/examples/webserver/concurrency-limits/src/main/resources/application.yaml new file mode 100644 index 00000000..0e9d72bb --- /dev/null +++ b/examples/webserver/concurrency-limits/src/main/resources/application.yaml @@ -0,0 +1,37 @@ +# +# Copyright (c) 2024 Oracle and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +server: + port: 8080 + host: 0.0.0.0 + concurrency-limit: + # Default port uses fixed limit algorithm + fixed: + permits: 6 + queue-length: 4 + queue-timeout: PT10S + sockets: + - name: "aimd" + port: 8088 + host: 0.0.0.0 + concurrency-limit: + # Second port uses aimd limit algorithm + aimd: + min-limit: 6 + initial-limit: 6 + max-limit: 10 + backoff-ratio: 0.5 + timeout: PT3S \ No newline at end of file diff --git a/examples/webserver/concurrency-limits/src/main/resources/logging.properties b/examples/webserver/concurrency-limits/src/main/resources/logging.properties new file mode 100644 index 00000000..1fe67722 --- /dev/null +++ b/examples/webserver/concurrency-limits/src/main/resources/logging.properties @@ -0,0 +1,22 @@ +# +# Copyright (c) 2024 Oracle and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +handlers=io.helidon.logging.jul.HelidonConsoleHandler +java.util.logging.SimpleFormatter.format=%1$tY.%1$tm.%1$td %1$tH:%1$tM:%1$tS.%1$tL %5$s%6$s%n +# Global logging level. Can be overridden by specific loggers +.level=INFO + +#io.helidon.examples.webserver.ratelimit.SleepService.level=FINE diff --git a/examples/webserver/concurrency-limits/src/test/java/io/helidon/examples/webserver/concurrencylimits/MainTest.java b/examples/webserver/concurrency-limits/src/test/java/io/helidon/examples/webserver/concurrencylimits/MainTest.java new file mode 100644 index 00000000..c80fcea4 --- /dev/null +++ b/examples/webserver/concurrency-limits/src/test/java/io/helidon/examples/webserver/concurrencylimits/MainTest.java @@ -0,0 +1,49 @@ +/* + * Copyright (c) 2024 Oracle and/or its affiliates. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package io.helidon.examples.webserver.concurrencylimits; + +import io.helidon.http.Status; +import io.helidon.webclient.api.HttpClientResponse; +import io.helidon.webclient.api.WebClient; +import io.helidon.webserver.http.HttpRouting; +import io.helidon.webserver.testing.junit5.ServerTest; +import io.helidon.webserver.testing.junit5.SetUpRoute; +import org.junit.jupiter.api.Test; + +import static org.hamcrest.CoreMatchers.is; +import static org.hamcrest.MatcherAssert.assertThat; + +@ServerTest +class MainTest { + private final WebClient client; + + protected MainTest(WebClient client) { + this.client = client; + } + + @SetUpRoute + static void routing(HttpRouting.Builder builder) { + Main.fixedRouting(builder); + } + + @Test + void testSleep() { + try (HttpClientResponse response = client.get("fixed/sleep/1").request()) { + assertThat(response.status(), is(Status.OK_200)); + } + } +} \ No newline at end of file diff --git a/examples/webserver/pom.xml b/examples/webserver/pom.xml index 1e75eb90..75feadef 100644 --- a/examples/webserver/pom.xml +++ b/examples/webserver/pom.xml @@ -36,6 +36,7 @@ basic basics comment-aas + concurrency-limits echo fault-tolerance grpc