History

Classify ECONNREFUSED as a retryable storage connection fault

Rhys Sullivan committed 2 months ago

Commit 49a351f1 parent bf5263f3 files · +11 −3Browse files

Unified · Split

3 files
  1. .changeset/storage-error-shaping.md +1 −1
  2. packages/core/sdk/src/fuma-runtime.test.ts +5 −0
  3. packages/core/sdk/src/fuma-runtime.ts +5 −2
.changeset/storage-error-shaping.md+1−1
@@ -5,4 +5,4 @@ · 4 unmodified lines
5 5
6Build `StorageError.message` from the call-site label plus the driver's error code instead of the driver's raw text. The driver text is drizzle's `Failed query: <sql>\nparams: <bound values>`, so error reporting grouped one storage defect by statement shape and printed bound parameters into issue titles. The full driver error stays on `cause`.6Build `StorageError.message` from the call-site label plus the driver's error code instead of the driver's raw text. The driver text is drizzle's `Failed query: <sql>\nparams: <bound values>`, so error reporting grouped one storage defect by statement shape and printed bound parameters into issue titles. The full driver error stays on `cause`.
7 7
8Add `StorageConnectionError`, a `StorageFailure` variant for postgres.js connection faults (`CONNECTION_ENDED`, `CONNECTION_CLOSED`, `CONNECTION_DESTROYED`, `CONNECT_TIMEOUT`, `ECONNRESET`) and workerd's cross-request I/O rejection. It carries the fault `code` and a `retryable` flag so a lost socket can be told apart from a pool-lifetime bug.8Add `StorageConnectionError`, a `StorageFailure` variant for postgres.js connection faults (`CONNECTION_ENDED`, `CONNECTION_CLOSED`, `CONNECTION_DESTROYED`, `CONNECT_TIMEOUT`, `ECONNREFUSED`, `ECONNRESET`) and workerd's cross-request I/O rejection. It carries the fault `code` and a `retryable` flag so a lost socket can be told apart from a pool-lifetime bug.
packages/core/sdk/src/fuma-runtime.test.ts+5−0
@@ -143,6 +143,11 @@ describe("fumaFailureFromCause — classification", () => { · 142 unmodified lines
143 ["CONNECTION_CLOSED"],143 ["CONNECTION_CLOSED"],
144 ["CONNECTION_DESTROYED"],144 ["CONNECTION_DESTROYED"],
145 ["CONNECT_TIMEOUT"],145 ["CONNECT_TIMEOUT"],
146 // A backend that is simply down: postgres.js surfaces the socket errno
147 // verbatim, so `connect ECONNREFUSED …` is the ordinary "database
148 // unreachable" fault and must classify like the rest.
149 ["ECONNREFUSED"],
150 ["ECONNRESET"],
146 ])("classifies postgres.js %s as a storage connection fault", (code) => {151 ])("classifies postgres.js %s as a storage connection fault", (code) => {
147 const failure = fumaFailureFromCause(152 const failure = fumaFailureFromCause(
148 "plugin_storage.findFirst",153 "plugin_storage.findFirst",
packages/core/sdk/src/fuma-runtime.ts+5−2
@@ -19,8 +19,10 @@ export class UniqueViolationError extends Data.TaggedError("UniqueViolationError · 18 unmodified lines
19 * bug that must stay loud.19 * bug that must stay loud.
20 *20 *
21 * `retryable` says which. It is a property of the fault, not a policy:21 * `retryable` says which. It is a property of the fault, not a policy:
22 * - `true` — the socket died underneath a live pool (`CONNECTION_CLOSED`,22 * - `true` — the socket died underneath a live pool, or was never
23 * `CONNECT_TIMEOUT`, `ECONNRESET`). Reconnecting can succeed.23 * established because the backend was unreachable (`CONNECTION_CLOSED`,
24 * `CONNECT_TIMEOUT`, `ECONNREFUSED`, `ECONNRESET`). Reconnecting can
25 * succeed.
24 * - `false` — the pool was already torn down or the socket belongs to a26 * - `false` — the pool was already torn down or the socket belongs to a
25 * different request context (`CONNECTION_ENDED`, `CONNECTION_DESTROYED`,27 * different request context (`CONNECTION_ENDED`, `CONNECTION_DESTROYED`,
26 * the workerd cross-request I/O rejection). Retrying is futile by28 * the workerd cross-request I/O rejection). Retrying is futile by
@@ -94,6 +96,7 @@ const isUniqueViolation = (cause: unknown): boolean => { · 67 unmodified lines
94const RETRYABLE_CONNECTION_CODES: ReadonlySet<string> = new Set([96const RETRYABLE_CONNECTION_CODES: ReadonlySet<string> = new Set([
95 "CONNECTION_CLOSED",97 "CONNECTION_CLOSED",
96 "CONNECT_TIMEOUT",98 "CONNECT_TIMEOUT",
99 "ECONNREFUSED",
97 "ECONNRESET",100 "ECONNRESET",
98]);101]);
99const FATAL_CONNECTION_CODES: ReadonlySet<string> = new Set([102const FATAL_CONNECTION_CODES: ReadonlySet<string> = new Set([