The Pipeline That Reported Success While Three Channels Went Dark

The brief

The operator noticed something a dashboard never would: the posts weren’t there. Not an alert, not a red light — just an absence, spotted by a human scrolling his own feed and thinking hang on.

Everything responsible for noticing said things were fine. The orchestrator recorded the run as status=success. The health watchdog wrote Scheduled Post Pipeline OK every hour, sixteen times, on top of a pipeline that had already stopped delivering. The queue file said the day’s post was handled.

This is the expensive failure mode in automation, and it is not the one people buy monitoring for. A crashed job pages somebody. A job that completes, returns success, and quietly does two-thirds of its work will run that way until a person happens to look — and the more reliable the system has been, the longer nobody looks.

What we found

Three defects, stacked, each individually reasonable.

A vendor deleted an API during a rebrand. The image generator the pipeline called had been folded into a new brand, and the old workflow-run endpoint was removed — not turned off, not deprecated with a sunset header, just gone, answering with an HTML 404 that an HTTP node happily accepted as a response. The domain now redirects to the new brand, and the redirect target 404s too. There was no changelog, no status page, no email. This was the second silent host migration from the same vendor; the first one is written down in our own notes, with the instruction to probe both hosts before blaming the key. That note is what made the diagnosis fast.

A quality gate discarded the thing it was protecting. A vision check exists because months earlier a generated image put a human in a company shirt and shipped it, unseen, to three platforms. So now every candidate image gets inspected, and a human in the frame is a hard reject. Correct. But on rejection the gate set the image to null and let the run continue — and two of the four destinations physically cannot publish without media. The gate was written to prevent a bad post. It was not written to consider what happens to the other channels when it fires, so a quality miss on one asset became silence on two platforms. It fired, regenerated twice, gave up, and darkened them.

The orchestrator graded itself on the wrong question. The publish step returned success: true alongside instagramSuccess: false and tiktokSuccess: false in the same object. It had answered “did I finish?” instead of “did the work arrive?” Everything downstream — the run status, the health watchdog, the state file — inherited that answer and repeated it.

The ground truth took one query against the publishing API, counting what actually shipped per platform per day: the day before, 3 TikTok / 4 Instagram / 5 X. The day of, 0 / 1 / 2 — and the single Instagram post was unrelated content that took a different path. That is the number that ends the argument, and no internal log contained it.

What we built

The gate now heals instead of darkening. It still hard-rejects a human — that rule is not up for negotiation, it exists because of a real incident — but when it cannot clear a candidate it falls back to a known-safe brand asset so the media-required channels keep publishing, and it records the miss loudly instead of passing a null downstream. Rejecting one image is not a reason to lose a day of distribution on two platforms.

The same audit found the reverse failure on another channel: comment replies had been recorded as permanently impossible, because the platform’s read API is genuinely gated behind a partner program. Only the read was gated. The write had been available the whole time, and the platform emails every comment to the post author — a mailbox already wired into the stack. One true sentence about one direction had been filed as a permanent verdict on both. Ten real comments, the oldest twenty-four days old, went from unanswerable to answered that afternoon.

What it means for you

Most AI-automation work sold today is construction: someone builds you a pipeline, hands it over, and it works on the day of the demo. The interesting question arrives six weeks later, when a vendor changes something without telling you, a gate fires on an edge case nobody modelled, and your dashboard keeps showing green because it is reading the same optimistic field the broken step wrote.

That is the work here. Read what the system actually did rather than what it says it did. Probe capabilities separately instead of trusting a blanket “not supported.” Check the failure against ground truth outside the system’s own reporting. Then make the guard heal rather than merely refuse.

We know the shape of these because we run a fleet of them in public and write down every way it has lied to us.

This engagement is now productized: the Silent Failure Audit runs the same failure archaeology against your stack — fixed scope, one week, findings with receipts. The free Silent Failure Checklist is the first pass you can run yourself.