Fixes #12558 - Document graceful shutdown.

Added documentation about graceful shutdown and `GracefulHandler`.

Moved property `jetty.server.stopTimeout` from `jetty.xml` to `jetty-graceful.xml`.

Signed-off-by: Simone Bordet <[email protected]>

documentation/jetty/modules/code/examples/src/main/java/org/eclipse/jetty/docs/programming/server/http/HTTPServerDocs.java

@@ -84,6 +84,7 @@
 import org.eclipse.jetty.server.handler.CrossOriginHandler;
 import org.eclipse.jetty.server.handler.DefaultHandler;
 import org.eclipse.jetty.server.handler.EventsHandler;
+import org.eclipse.jetty.server.handler.GracefulHandler;
 import org.eclipse.jetty.server.handler.QoSHandler;
 import org.eclipse.jetty.server.handler.ResourceHandler;
 import org.eclipse.jetty.server.handler.SecuredRedirectHandler;
@@ -1637,6 +1638,37 @@ public void defaultHandler() throws Exception
         // end::defaultHandler[]
     }
 
+    public void gracefulHandler() throws Exception
+    {
+        // tag::gracefulHandler[]
+        Server server = new Server();
+
+        // Install the GracefulHandler.
+        GracefulHandler gracefulHandler = new GracefulHandler();
+        server.setHandler(gracefulHandler);
+
+        // Set the Server stopTimeout to wait at most
+        // 10 seconds for existing requests to complete.
+        server.setStopTimeout(10_000);
+
+        // Add one web application.
+        class MyWebApp extends Handler.Abstract
+        {
+            @Override
+            public boolean handle(Request request, Response response, Callback callback) throws Exception
+            {
+                // Implement your web application.
+                callback.succeeded();
+                return true;
+            }
+        }
+        ContextHandler contextHandler = new ContextHandler(new MyWebApp(), "/app");
+        gracefulHandler.setHandler(contextHandler);
+
+        server.start();
+        // end::gracefulHandler[]
+    }
+
     public void continue100()
     {
         // tag::continue100[]

documentation/jetty/modules/operations-guide/pages/modules/standard.adoc

@@ -168,6 +168,19 @@ Jetty's configuration properties are identical across all versions of this modul
 include::{jetty-home}/modules/ee10-webapp.mod[tags=ini-template]
 ----
 
+[[graceful]]
+== Module `graceful`
+
+The `graceful` module allows to shut down gracefully the Jetty server when it is stopped (see xref:start/index.adoc#stop[this section] for more information about stopping Jetty).
+
+The `graceful` module installs the `GracefulHandler` at the root of the `Handler` tree; the `GracefulHandler` rejects new requests, but allows current requests to terminate within a configurable timeout, as explained in xref:programming-guide:server/http.adoc#handler-use-graceful[this section].
+
+The module properties are:
+
+----
+include::{jetty-home}/modules/graceful.mod[tags=documentation]
+----
+
 [[http]]
 == Module `http`
 

documentation/jetty/modules/operations-guide/pages/start/index.adoc

@@ -540,6 +540,18 @@ The `jetty.server.stopAtShutdown` property configures a JVM shutdown hook that i
 
 Obviously, the JVM can also be stopped with `kill -KILL <pid>` that exits the process abruptly without running the JVM shutdown hooks.
 
+[[stop-graceful]]
+=== Stopping Gracefully
+
+Stopping Jetty abruptly when there are active HTTP requests being handled may result in a variety or errors, because Jetty components that are used to handle requests are being stopped concurrently.
+
+For example, when the Jetty thread pool is stopped, an attempt to submit a task would throw `RejectedExecutionException`; when a component is stopped, its fields may be nulled-out, resulting in a `NullPointerException` being thrown if the component is used; etc.
+
+You can stop Jetty _gracefully_ by adding the `graceful` Jetty module (see xref:modules/standard.adoc#graceful[this section] for more information).
+
+When Jetty is stopped, the `graceful` module organizes to reject new requests, but allows existing requests to finish within a configurable timeout; then closes all the connections, stops all the Jetty components, and then exits the JVM.
+In this way, existing requests are not responded with an error caused by the server stopping, provided they complete within the timeout.
+
 [[stop-remote]]
 === Stopping Jetty from Remote
 

documentation/jetty/modules/programming-guide/pages/server/http.adoc

@@ -1254,6 +1254,41 @@ The `Forwarded` header is typically present in requests that have been forwarded
 
 Note that `ThreadLimitHandler` is different from xref:handler-use-qos[`QoSHandler`] in that it limits the number of concurrent requests per remote IP address, while `QoSHandler` limits the total number of concurrent requests.
 
+[[handler-use-graceful]]
+==== GracefulHandler
+
+`GracefulHandler` allows to stop the Jetty server in a graceful way, by rejecting new requests but allowing existing requests to complete within a configurable timeout.
+
+In this way, existing requests can be completed normally rather than with failures caused by the fact that the server is stopping; for example, stopping the server causes all TCP connections to be closed, so trying to write a response will result in a failure.
+
+In order to stop Jetty gracefully, you need the following:
+
+* Install the `GracefulHandler`, typically just after the `Server` at the root of the `Handler` tree.
+* Configure `Server.stopTimeout` to a positive value.
+
+When the `Server` component is stopped, it will check the `Server.stopTimeout`, and if positive, it will initiate a graceful shutdown by notifying all components that implement the `Graceful` interface that the shutdown has been initiated.
+
+`GracefulHandler` implements `Graceful`, so it will start rejecting new requests with status code `503 Service Available`, but will allow existing requests to complete for a period of time up to `Server.stopTimeout`.
+
+When all existing requests have completed, the `Server` stops all ``Connector``s, closes all connections, and finally stops all the components.
+
+This is how you configure and use `GracefulHandler`:
+
+[,java,indent=0]
+----
+include::code:example$src/main/java/org/eclipse/jetty/docs/programming/server/http/HTTPServerDocs.java[tags=gracefulHandler]
+----
+
+The `Handler` tree structure looks like the following:
+
+[,screen]
+----
+Server
+  └── GracefulHandler
+      └── ContextHandler
+          └── MyWebApp
+----
+
 [[handler-use-servlet]]
 === Servlet API Handlers
 

jetty-core/jetty-server/src/main/config/etc/jetty-graceful.xml

@@ -11,4 +11,5 @@
       <New id="GracefulHandler" class="org.eclipse.jetty.server.handler.GracefulHandler" />
     </Arg>
   </Call>
+  <Set name="stopTimeout"><Property name="jetty.server.stopTimeout" default="5000"/></Set>
 </Configure>

jetty-core/jetty-server/src/main/config/etc/jetty.xml

@@ -90,7 +90,6 @@
     <!-- extra server options                                        -->
     <!-- =========================================================== -->
     <Set name="stopAtShutdown"><Property name="jetty.server.stopAtShutdown" default="true"/></Set>
-    <Set name="stopTimeout"><Property name="jetty.server.stopTimeout" default="5000"/></Set>
     <Set name="dumpAfterStart" property="jetty.server.dumpAfterStart"/>
     <Set name="dumpBeforeStop" property="jetty.server.dumpBeforeStop"/>
     <Set name="tempDirectory" property="jetty.server.tempDirectory"/>

jetty-core/jetty-server/src/main/config/modules/graceful.mod

@@ -11,3 +11,10 @@ server
 
 [xml]
 etc/jetty-graceful.xml
+
+[ini-template]
+# tag::documentation[]
+## The timeout, in milliseconds, to apply when stopping the server gracefully.
+# jetty.server.stopTimeout=5000
+# end::documentation[]
+

jetty-core/jetty-server/src/main/config/modules/server.mod

@@ -89,9 +89,6 @@ etc/jetty.xml
 ## Whether ctrl+c on the console gracefully stops the Jetty server
 # jetty.server.stopAtShutdown=true
 
-## Timeout in ms to apply when stopping the server gracefully
-# jetty.server.stopTimeout=5000
-
 ## Dump the state of the Jetty server, components, and webapps after startup
 # jetty.server.dumpAfterStart=false