Every workflow on our box that fails routes to a single error-handling workflow, error-alert, which sends a Telegram alert within seconds. Until this week that was all it did: a workflow threw an error, error-alert fired, and Telegram got a message. We left the wiring into it alone and rebuilt what happens in between.
Alerting on everything is reasonable when every failure is worth a human's attention, and less so once unattended workflows fail for reasons no human can act on: a socket hang-up mid-request, or an API that times out and recovers a few seconds later.
error-alert (internal ID marainErrorAlert) is what twenty-two of the other twenty-three workflows on the box name in settings.errorWorkflow. The upgrade happened inside that same workflow ID rather than in a replacement, so none of those references changed, and every path through the new version still ends at the same Send Telegram alert node the original used, with different message text.
Triage before alert
On a failure, the workflow gathers the execution context (which workflow, which node, the error text, the execution ID) and sends it to a local model, qwen2.5:7b-instruct on our Mac Studio, asking for three fields: a category of TRANSIENT, LOGIC or UNKNOWN, a confidence score, and a one-sentence reason. Three things can go wrong at that step: the environment variable holding the endpoint is unset, the endpoint is unreachable, or the reply is not parseable JSON. All three produce the same output, category UNKNOWN at confidence 0, which can never pass the retry gate. A broken triage step leaves the workflow behaving as it did before this build.
A retry happens only when three conditions hold together: the triage classified the failure as transient, its confidence is at least 0.7, and the failed workflow is on an explicit retry-safe allowlist. Retries then run against a durable budget of two attempts with exponential backoff, tracked in an n8n Data Table row keyed by workflow ID, so a container restart does not reset the count. Anything that misses one of the three conditions, or that has spent its budget, escalates the way everything used to.
The allowlist has one entry
The list of workflows allowed to retry is the part of this build we kept most conservative. It ships with one entry, social-listening, which runs on a schedule, reads three public APIs, writes nothing to the CRM, and dedupes its own Telegram digest against previously-seen URLs, so a repeated run costs a duplicate digest at worst.
A workflow earns a place on the list only by meeting four conditions together: it makes no outward sends to a third party, it behaves idempotently on a repeated run, it has an executeWorkflowTrigger entry point (which is also how the retry fires), and it runs on a schedule rather than holding open a live webhook connection. cal-booking, tool-capture and anything else that messages someone outside the firm fail the first condition and cannot be added. A confident transient classification cannot put a workflow on the list.
We checked that the allowlist and the triage gate decide independently. A throwaway test workflow, temporarily added to the allowlist, threw a transient-styled failure. It triaged as TRANSIENT at 0.9 confidence, retried twice with measured waits of 30 and 121 seconds, and escalated once the two-attempt budget was spent. The same workflow then threw a TypeError, which triaged as LOGIC at confidence 0 and was never retried: six nodes ran instead of fifteen, and no retry execution followed.
The idempotency gate
We also built idempotency-gate, a seven-node sub-workflow any webhook workflow can call before acting on a request. It takes a key, checks an n8n Data Table for a prior sighting of it, and reports whether this is a duplicate. Called twice with the same key, the first call returned duplicate: false and wrote a row, and the second returned duplicate: true with the original timestamp unchanged, which is how we know no second write happened. Nothing calls it yet: cal-booking already carries dedup logic of its own, tool-capture has none, and changing live workflows was out of scope here.
What live runs found
Two bugs surfaced only once we ran real executions, and workflow validation reported both paths clean. First, n8n skips a node entirely when its input has zero items, so a workflow's very first failure, with no retry-budget row yet, produced no retry and no Telegram message at all. Setting alwaysOutputData on the lookup node forces a placeholder item through and fixed it, the same behavior we hit in the tenant resolver. Second, the Wait node read its backoff duration from $json, which by then held the row the upstream Data Table upsert had just written rather than the context flowing in; referencing the earlier node by name fixed that. Both bugs run through a Data Table node's output, and neither is the kind of thing static validation can see.
Net effect
Before this build, every failure produced one identical Telegram message. Now a transient failure on the one allowlisted workflow retries itself with real backoff and, if it recovers, never reaches a human. Everything else escalates as it did, with one added line saying what was tried and why. If you are running unattended automation and want to work out which of your own failures are worth a page, that is one of the things the audit is for.