feat(redis): add reconnectOnError for READONLY / LOADING reply errors

Empirical proof from the redis-failover-harness (TRI-8878) shows
ElastiCache vertical scale-up events surface as Redis-level reply
errors (READONLY when the role swap happens under an open connection,
LOADING when a node is still initializing) rather than connection-level
errors. Without intervention these errors propagate directly to caller
code at the rate of tens of thousands per minute over a multi-minute
window.

Returning 2 from reconnectOnError tells ioredis to tear down the
connection, reconnect, and re-issue the failed command. After
reconnect, DNS / SG state routes the new socket to a writable node and
the workload resumes.

Harness measurements:
- Without this option: ~437,000 caller-surfaced errors over 4 min of
  zero-write throughput per task during cache.t4g.medium -> m7g.large.
- With this option (same workload, m7g.large -> m7g.xlarge): 2 total
  caller-surfaced errors across both tasks; throughput uninterrupted.

Reduction: ~218,000x.

Scope: only the shared createRedisClient helper in @internal/redis.
Direct 'new Redis()' callsites in apps/webapp/ still need migration;
defaultReconnectOnError is exported so they can opt in inline as a
follow-up.

Refs TRI-8868 TRI-8878.

Author: ericallam

.server-changes/redis-reconnect-on-readonly-loading.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Add `reconnectOnError` to the shared ioredis client config so READONLY / LOADING reply errors during ElastiCache node-type changes trigger a disconnect-reconnect-retry cycle instead of surfacing to caller code.

internal-packages/redis/src/index.ts

@@ -3,12 +3,32 @@ import { Logger } from "@trigger.dev/core/logger";
 
 export { Redis, type Callback, type RedisOptions, type Result, type RedisCommander } from "ioredis";
 
+/**
+ * Reply-error -> reconnect mapping. Without this hook, an ElastiCache
+ * vertical scale-up surfaces tens of thousands of READONLY / LOADING
+ * reply errors to caller code over a healthy TCP/TLS connection (the
+ * client keeps talking to a node whose role swapped underneath it).
+ *
+ * Returning 2 tells ioredis to disconnect, reconnect, and retry the
+ * command that triggered the error. After reconnect, DNS / SG routing
+ * should land on a writable primary.
+ *
+ * Empirical confirmation on the harness in TRI-8878: this option
+ * reduced a scale-up event from ~437,000 caller-surfaced errors to 2.
+ */
+export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
+  const msg = err.message ?? "";
+  if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
+  return false;
+}
+
 const defaultOptions: Partial<RedisOptions> = {
   retryStrategy: (times: number) => {
     const delay = Math.min(times * 50, 1000);
     return delay;
   },
   maxRetriesPerRequest: process.env.GITHUB_ACTIONS ? 50 : process.env.VITEST ? 5 : 20,
+  reconnectOnError: defaultReconnectOnError,
 };
 
 const logger = new Logger("Redis", "debug");