When the monitor cannot read

A monitor has two jobs. The first is to tell you when something is wrong. The second, which gets far less attention, is to tell you when it cannot tell.

Picture a reconciliation check. It reads a share count, a price and a cash balance, multiplies and adds, and compares the result with a published value. One night the price source times out. The code catches the exception, logs it, and moves on. No discrepancy was computed, so no discrepancy was reported, and the dashboard stays green.

Green that means “I didn’t look” is the most expensive colour in operations. Everyone downstream reads it as “we looked, and it was fine”.

Why it happens

Most checks are written as booleans: the condition held, or it did not. Data errors do not fit either branch, so they end up in a log file. Add alert fatigue, where noisy read failures get suppressed so the real alerts stand out, and a monitor can be silent for exactly the reason it was built to catch.

Five states, not two

The fix starts with vocabulary. Every check should end in one of these states:

• PASSEvery input was read, fresh, and the assertion held.
• FAILInputs were trustworthy and the assertion was violated.
• STALEAn input was read but is older than its stated cadence.
• UNAVAILABLEA source could not be read. Never reported as healthy.
• INDETERMINATEData was read but cannot settle the question, for example across mismatched cutoffs.

The distinctions are not pedantry. A stale input should never produce a FAIL, because that accuses the published number on the strength of evidence you do not trust. Two values read at different block heights should not be compared at all, because the difference may be timing rather than loss. That is INDETERMINATE, and saying so is more useful than a false alarm or a false all-clear.

What a check should say

An alert is only actionable if the person receiving it can verify it without re-running the check. At minimum, it should carry:

check        nav.reconcile
state        FAIL
expected     13,396,412.99   shares × price + cash
published    13,512,870.00
delta        −116,457.01     (−0.86%, tolerance ±0.05%)
cutoff       block 24,118,302
inputs       shares  on-chain     age 0 blocks
             price   feed         age 2h   (cadence 24h)
             cash    on-chain     age 0 blocks
owner        valuation
next         escalate before next settlement window

Sources, cutoffs, input ages, the arithmetic and the tolerance are what turn a red light into a decision. The owner and next action are what turn a decision into a response.

What the system does when it cannot read

• Retry within a bound, then stop pretending: report UNAVAILABLE.
• Fall back to a second source only if it is genuinely independent, and say in the output that a fallback was used.
• Never carry a last-known value forward silently. A cached price is a stale price with better manners.
• Escalate an UNAVAILABLE once it lasts longer than the cadence of the thing it protects. A missed hourly read matters less than a missed read before a monthly settlement.
• Monitor the monitor. A heartbeat that stops is itself an alert, routed to someone other than the monitor.

Tolerances need reasons

Every threshold should be derivable: rounding in share decimals, expected price movement between update intervals, fee accrual between cutoffs. A tolerance chosen because it makes the alert stop firing is not a tolerance. It is a decision to stop looking, made by default.

What this does not solve

If every source you reconcile against ultimately repeats the same upstream assertion, their agreement proves very little. Independence is a property to verify, not assume. And a monitor, however careful, is not authority: someone still has to own the decision it informs.

The hard part of operational assurance is rarely the arithmetic. It is agreeing what the number means, where it comes from, who acts on it, and what happens on the night the data does not arrive.