Readiness and draining traffic

When SIGTERM arrives, the service should stop reporting ready before it finishes active requests. Load balancers can then move traffic elsewhere while the instance drains.

How do I drain connections before shutdown?

Use this sequence:

1. Set an application-wide draining state.
2. Make the readiness route return a non-ready status.
3. Reject new application work.
4. Let requests that started earlier finish.
5. Close the listener and shared resources.
6. Exit before the platform deadline.

Readiness changes are not immediate everywhere. A load balancer may poll only every few seconds, and endpoint updates take time to propagate. The service must handle requests that arrive during that delay.

Choose a drain pattern

Both can pass shutdown-check. Returning 503 provides a clearer response to clients and allows a short window for traffic systems to update.

Build a server that drains

const http = require("node:http");
 
const port = Number(process.env.PORT ?? 3000);
const drainDelayMs = Number(process.env.DRAIN_DELAY_MS ?? 1000);
let draining = false;
 
const server = http.createServer((request, response) => {
  if (request.url === "/health") {
    response.writeHead(draining ? 503 : 200, {
      "content-type": "text/plain",
    });
    response.end(draining ? "draining" : "ok");
    return;
  }
 
  if (draining) {
    response.writeHead(503, {
      connection: "close",
      "retry-after": "1",
    });
    response.end("shutting down");
    return;
  }
 
  if (request.url === "/slow") {
    response.writeHead(200, { "content-type": "text/plain" });
    response.flushHeaders();
    setTimeout(() => response.end("work complete\n"), 2000);
    return;
  }
 
  if (request.url === "/test/new-work") {
    response.writeHead(200).end("accepted");
    return;
  }
 
  response.writeHead(404).end();
});
 
server.listen(port, "127.0.0.1");
 
process.on("SIGTERM", () => {
  if (draining) return;
  draining = true;
 
  setTimeout(() => {
    server.close((error) => {
      if (error) {
        console.error(error);
        process.exitCode = 1;
      }
    });
  }, drainDelayMs);
});

Requests already inside a handler do not check draining again, so they finish normally. New requests pass through the draining guard and receive 503.

The delay keeps the listener available while traffic systems observe the readiness change. After that, server.close() stops connections and waits for the original work.

Configure the drain checks

{
  "command": ["node", "server.js"],
  "baseUrl": "http://127.0.0.1:3000",
  "readiness": { "path": "/health", "status": 200 },
  "workload": {
    "path": "/slow",
    "bodyIncludes": "work complete",
    "concurrent": 2,
    "started": { "type": "response-headers" }
  },
  "shutdown": {
    "deadlineMs": 10000,
    "readinessWithdrawal": true,
    "newRequests": {
      "path": "/test/new-work",
      "rejectStatuses": [503],
      "allowConnectionRefused": true
    },
    "repeatSignalAfterMs": 500
  }
}

What readinessWithdrawal checks

After SIGTERM, shutdown-check polls the normal readiness route. The check passes when it sees:

• any status other than readiness.status; or
• a refused or reset connection.

A request timeout does not count. It does not clearly tell a load balancer that the service is unavailable.

If readiness stays ready until the deadline, shutdown-check returns SC310.

What newRequests checks

After readiness is withdrawn, shutdown-check sends one GET to newRequests.path while the original workload remains active.

The new request must:

• return a status in rejectStatuses; or
• be refused when allowConnectionRefused is true.

A timeout, 200, or another unlisted status returns SC311. If the original work ends before the new request can be tested, the result is SC312.

Choose a safe GET route. shutdown-check really calls it, so it must not create orders, send email, or change production data.

Read a passing timeline

PASS SC000: Graceful shutdown verified
 
Timeline:
  +    5 ms  process launched — pid=73882
  +  112 ms  service ready — HTTP 200
  +  112 ms  work request sent — #1 GET /slow
  +  112 ms  work request sent — #2 GET /slow
  +  113 ms  work confirmed active — 2 response(s) sent headers; bodies still in progress
  +  113 ms  signal sent — SIGTERM
  +  114 ms  readiness withdrawn — HTTP 503
  +  115 ms  new request rejected — HTTP 503
  +  615 ms  signal repeated — SIGTERM
  + 2115 ms  work request finished — #1 HTTP 200
  + 2115 ms  work request finished — #2 HTTP 200
  + 2118 ms  process exited — code=0, signal=none
  + 2119 ms  shutdown verified — work completed and service exited before deadline

The key ordering is readiness withdrawal and new-request rejection before the original work finishes.

Choose the drain delay

Start with the load balancer's readiness polling interval multiplied by its failure threshold. Add time for endpoint propagation where relevant.

The total must still fit:

drain delay + slowest active request + cleanup < platform shutdown deadline

Set shutdown.deadlineMs to cover the server-side portion with a safety margin. On Kubernetes, remember that preStop time is included in terminationGracePeriodSeconds.

Handle repeated SIGTERM safely

Signal handlers should be idempotent. The first signal starts the drain; later signals should not restart timers, call cleanup twice, or force active work to exit.

The if (draining) return guard in the example handles this. Enable repeatSignalAfterMs to test the behavior.

Common failures

Related

• Graceful shutdown in Node.js
• Kubernetes and containers
• Shutdown configuration
• SC310: Readiness not withdrawn
• SC311: New request accepted