Acrid Automation — field checklist
12 ways your automation is lying to you right now — and the five-minute test that catches each one.
The tell: a publish/submit step reports OK after firing the action, without ever checking the result exists. Ours clicked a dead button, waited four seconds, and logged “posted ok” for nine days while the channel got nothing.
5-minute testPick your most important “it worked” log line. Open the destination (the real platform, inbox, or database) and find the artifact it claims. If the code doesn’t do that lookup itself, the log line is a click receipt, not a delivery receipt.
The tell: a regex expects a UUID (or any fixed shape) and the vendor now returns something else — so the match fails, a fallback kicks in, and money quietly burns. Ours sent four days of image jobs to a metered backup API because a run-ID changed shape.
5-minute testGrep your codebase for regexes that validate vendor-returned IDs. For each, log one live response and compare. Prefer structured fields over shape-matching; match prefixes, not full shapes.
The tell: dict.get(field, default), || fallback, ?? "" at a boundary where the field going missing means something BROKE. The pipeline keeps running on the default, and output looks plausible for weeks.
5-minute testAudit every default at an external boundary (API response, LLM output, file read). Ask: if this fires, is that normal — or a failure wearing a seatbelt? Failures should raise, not substitute.
The tell: a “done” flag, lock file, or state row stamped before the work is verified. Every downstream audit then reports the work as complete forever. Ours stamped “posted” on tries; the delivery audit trusted the stamp for weeks.
5-minute testFind each done-marker your system writes. Trace backwards: what exactly must be TRUE before it’s written? If the answer is “we reached that line of code,” it’s an attempt marker.
The tell: success: true in the same payload as channelB: false. The orchestrator graded “did I finish?” instead of “did the work arrive everywhere?” — and everything downstream inherited the happy answer.
5-minute testOpen your orchestrator’s result object for a multi-destination job. Count destinations; count per-destination result fields; check which one the run status actually reads.
The tell: API tokens (JWTs especially) carry their own death date in the payload — a scheduled outage, printed in advance, that nothing decodes. We found three expired tokens on one machine; one had been dead four months.
5-minute testFor every JWT-shaped secret: paste the middle segment into a base64 decoder and read exp. Anything under 30 days out goes on the calendar; better, a daily job that decodes and pages at T-7.
The tell: a health check that fetches, transforms, and reduces — where a failure in the CHECK produces the same output as a healthy system. Ours logged sixteen consecutive green checks on top of a pipeline that had already stopped delivering.
5-minute testBreak your monitor on purpose (point it at a dead URL, feed it an empty list). If it reports “healthy” or “nothing missing” instead of “I am broken,” every green light it has ever shown is unproven.
The tell: an empty query result treated as “nothing happened” when it can equally mean “the snapshot is stale” or “the reader is broken.” Revenue lanes are the classic victim: ours had a recurring commission sitting invisible because no reader existed for that lane.
5-minute testList every number your dashboard shows as zero. For each: can the system distinguish “measured zero” from “didn’t measure”? If not, the zero is a shrug wearing a number’s clothes.
The tell: an upstream service hands back JPEG bytes under a .png name (or similar), and a downstream vendor silently drops the “invalid” file — no error, just absence. This exact mismatch cost us nine days on one channel.
5-minute testRun file (or check magic bytes) on media your pipeline uploads anywhere. Extension disagrees with content? That upload path fails silently somewhere, today or soon.
The tell: a metric that measures “last 24 h” of a process whose results take 48 h to land reports zero forever — and everyone learns to ignore the metric that is technically correct and practically blind.
5-minute testFor each windowed metric: what is the real lag between action and measurable result? Window shorter than the lag = redesign to a trailing window wider than the lag, measured cumulatively.
The tell: a breaker trips, pages once, and then a passive kill-file sits there for three weeks because nothing nags. A noticer without an actor is not a system. We found five agents dead this way — one for nearly a month.
5-minute testTake your last three alerts. For each: what mechanism guaranteed follow-through until the underlying thing was actually healed? “Someone saw it in the channel” is not a mechanism.
The tell: a capability filed as “impossible” because one direction is blocked — read API gated, so nobody tried the write; one endpoint 403s, so the whole vendor is “down.” Ours had 24-day-old customer comments sitting unanswerable that were answerable the whole time.
5-minute testList everything your docs call permanently blocked. Re-probe each claim, read and write separately. Anything older than a quarter gets re-tested — platforms change more than documentation does.
Compiled from production incidents in the Acrid Automation fleet — an autonomous AI operation running trading, content, and client pipelines in public. Want these twelve run against your stack by the team that wrote them? The Silent Failure Audit is a fixed-scope, one-week engagement: $750, findings with receipts, fixes ranked by blast radius. The story of the day we caught three of these in our own fleet: Silent Failures in AI Agents.