Improve resource exhaustion handling

[slinkydeveloper]

When users reach the limit of threads number, the JVM will fail with simply

java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

Is it possible to make this failure "nicer"? or give an indication if this is a real OOM or it hits the limit of number of threads in Linux? How other java server frameworks handle that?

[gauthamchandra]

We ran into this very issue earlier today in our internal load tests for a new workflow we're testing.

I believe the issue is because the HandlerRunner, by default, uses an unbounded thread pool to run your ctx.run() blocks.

As a result, under high concurrency with enough long running ctx.run() blocks on the same machine, you could run into this scenario.

I feel like the thread pool should be bounded and based off a heuristic that takes into account the amount of available memory and processors. I can try to put up a PR for this later today 😄

[slinkydeveloper]

@gauthamchandra as a workaround, you can change the thread pool you want to use, providing the HandlerRunner.Options in Endpoint.bind().

Indeed the cached thread pool works best when handlers are short lived: when you have handlers that take long (and don't suspend in between because of a high inactivity timeout/long executing tasks) this can easily lead to spawning many threads, thus OOMing...

If you're in Java 21+, I would just suggest to go with the virtual threads executor, as it works best for this exact use case here of short lived/long lived tasks mix.

For anything less than Java 21, I guess a reasonable solution would be to either use the cached thread pool (as the default), or use a fixed thread pool with a max concurrency that depends on your use case.

One problem with limiting concurrency in the thread pool is that it can lead to stall behaviors that aren't easy to observe: Where is my request being blocked? waiting in the thread pool?
Because Restate retries anyway, I'm thinking perhaps a better approach would be for the server itself to signal the exhaustion #579 and let restate retry later on, instead of queueing up in the executor. So ideally, an executor when it exhaust the available concurrency, should just fail, and let restate retry...
WDYT?

[slinkydeveloper]

After more thinking, I came up with the following:

• If virtual threads available (Java 21+) -> then just go with that.
• If virtual threads not available -> Policy along the lines of:

static int THREADS_PER_CORE = 8

int cores = Runtime.getRuntime().availableProcessors();
int maxThreads = cores * 8;

ExecutorService executor = new ThreadPoolExecutor(
    cores,                 
    maxThreads,              
    60L, TimeUnit.SECONDS,  
    new SynchronousQueue<>(),  // Direct handoff
    new ThreadPoolExecutor.AbortPolicy() // If full -> throws an exception -> Restate will retry
);

This policy is close to what Tomcat and other industry standard servers do.

[slinkydeveloper]

Hey @gauthamchandra wanna check this out? #582