Why One Failed Check Should Not Mean Downtime

A single failed check is evidence that one probe did not get the response it expected. It is not automatically proof that customers are down. Networks drop packets. Deploys restart processes. DNS caches go weird for a moment. Confirmation exists so those blips do not become public incidents.

What one failure can mean

• A real outage just started
• A transient network error between checker and target
• A brief deploy or restart
• A checker-side timeout that customers did not feel
• An assertion that is too strict for a rare but valid response

You cannot tell which one from a single sample. That is why incident verification exists.

Blip Budget

Your Blip Budget is how many consecutive failed checks you can afford before the uncertainty costs more than the noise. It depends on check interval and how fast customers feel pain.

Long intervals with high confirmation counts hide real outages. Short intervals with zero confirmation create alert fatigue. Pick both together.

A sensible default sequence

1. Probe fails.
2. Retry once quickly if your monitor supports it.
3. Enter a verifying state rather than paging immediately.
4. Open an incident after consecutive confirmations.
5. Require consecutive successes before auto-recovery.

Product references: Incident verification, Monitor states.

When one failure should still page

• You are validating a brand-new monitor and want to learn quickly (temporary)
• The path is so critical that even brief uncertainty is unacceptable and humans accept the noise
• You have independent confirmation from another system in the same minute

Those cases are exceptions. Document them so the rest of the system can stay calm.

Flapping

If a monitor oscillates between success and failure, confirmation alone is not enough. Fix the assertion, the timeout, or the underlying instability. Muting forever hides a real reliability problem.

Customer reports versus probes

Sometimes customers feel pain before monitors confirm. Believe them enough to investigate, even if the public component stays in verifying. Confirmation protects against false pages. It is not permission to gaslight a queue of support tickets.

Tie it back to uptime math

How you treat unverified failures changes published uptime. Decide whether only confirmed downtime counts. Then keep that rule next to any percentage you show. See How to Calculate Uptime Correctly.

For API paths, combine this with the Confirm Ladder in How to Monitor an API Without Creating Alert Noise.