Spatial Pipeline EngineeringMRV Automation

Failure Mode Catalog for Emissions Validation Gates

A validation gate that has never blocked anything is not a clean pipeline; it is an untested control. This catalogue collects the ways a gate fails while continuing to report success, within emissions data quality and validation gates in the MRV architecture and carbon accounting fundamentals stack. It is the companion to the cross-stage failure mode catalog for spatial MRV pipelines, narrowed to the validation layer itself.

The organising insight is that a gate has two ways to be wrong and only one of them is visible. It can block something it should have passed, which produces an incident, a conversation, and a fix. Or it can pass something it should have blocked, which produces nothing at all — no error, no alert, no signal — until a verifier finds the record months later. Every entry below is a variant of the second.

Ten gate failure modes grouped by where the gate breaks A grid of ten failure modes in three groups. The definition group covers a check that coerces instead of rejecting, a threshold fitted to observed data rather than physics, and a check whose classification as blocking or advisory is wrong. The coverage group covers a suite that runs only at ingestion, a check sampled rather than applied to every row, a gate skipped when a downstream job is behind, and a schema check that permits unexpected columns. The response group covers alerts muted after fatigue, quarantined records with no exit path, and a documented exception that outlives its justification. A panel notes that none of the ten produce an error, and that all ten leave the pipeline reporting success. Ten ways to pass something you should have blocked None of them raise. All of them leave the run green. Definition — the check is wrong 1 · coerces instead of rejecting 2 · threshold fitted to observed data 3 · blocking/advisory misclassified the gate does what it was told, and what it was told is wrong Coverage — it never ran 4 · suite only at ingestion 5 · sampled, not applied to all rows 6 · skipped when the queue is behind 7 · schema permits extra columns the data never met the check Response — nobody acted 8 · alerts muted after fatigue 9 · quarantine with no exit path 10 · exception outliving its reason the check worked and fired, and the finding went nowhere

Root Cause Analysis

The ten entries share three underlying causes, and naming the cause shortens the diagnosis considerably.

A gate that can adjust rather than refuse will eventually adjust. Coercion is the archetype: a schema that casts a string to a float, a unit converter that guesses when the unit is absent, a geometry repair that runs automatically on invalid input. Each is a small convenience that turns a detectable defect into an undocumented assumption, and the assumption then propagates silently. The rule that closes the whole class is that a gate has exactly two outcomes — pass, or refuse and quarantine — and no third option that quietly fixes things.

Coverage decays faster than definitions. Check definitions are reviewed; where and when they run rarely is. A suite attached to the ingestion stage stays attached while three new stages appear downstream of it, a sampling rate introduced for a performance problem outlives the problem, and a “skip validation when behind” switch added during an incident becomes permanent. None of these change any check’s logic, and all of them reduce what the checks see.

A finding with no owner is not a control. Gates 8 through 10 all have working checks that fired correctly. What failed was the response path — an alert channel nobody reads, a quarantine queue nobody drains, an exception granted for one period that nobody revisits. From a verifier’s standpoint these are indistinguishable from having no check at all, and they are more expensive because the organisation believed it was covered.

Diagnostic Pipeline / Pre-Flight Validation

Most of the catalogue is detectable by auditing the gate layer itself rather than the data. The checks below answer three questions: does every check actually run on everything, has any check ever fired, and is every finding resolved.

from dataclasses import dataclass
from datetime import date, timedelta

import structlog

log = structlog.get_logger()


@dataclass(frozen=True)
class GateHealth:
    check_id: str
    blocking: bool
    runs_in_period: int
    rows_evaluated: int
    rows_in_scope: int
    fired_ever: bool
    days_since_last_fire: int | None
    verdict: str


def audit_checks(stats: list[dict], expected_runs: int,
                 today: date) -> list[GateHealth]:
    """Audit the gates, not the data.

    Three questions per check: did it run everywhere it should have, did it see
    every row, and has it ever fired. A check that has never fired in two years
    is either describing something impossible or is miscalibrated — and both
    deserve investigation rather than comfort.
    """
    out = []
    for s in stats:
        coverage = s["rows_evaluated"] / max(s["rows_in_scope"], 1)
        last_fire = s.get("last_fired")
        days_since = (today - last_fire).days if last_fire else None

        verdict = "healthy"
        if s["runs_in_period"] < expected_runs:
            verdict = "under_run"
        elif coverage < 0.999:
            verdict = "sampled_or_partial"
        elif not s["fired_ever"] and s.get("age_days", 0) > 730:
            verdict = "never_fired_in_two_years"
        elif s["blocking"] and days_since is not None and days_since > 365:
            verdict = "dormant_blocking_check"

        health = GateHealth(
            check_id=s["check_id"], blocking=s["blocking"],
            runs_in_period=s["runs_in_period"], rows_evaluated=s["rows_evaluated"],
            rows_in_scope=s["rows_in_scope"], fired_ever=s["fired_ever"],
            days_since_last_fire=days_since, verdict=verdict,
        )
        if verdict != "healthy":
            log.warning("gate.audit", **health.__dict__)
        out.append(health)

    return out


def audit_dispositions(quarantine: list[dict], today: date,
                       max_age_days: int = 45) -> dict:
    """Every quarantined record must exit through a door.

    Corrected, excepted with an owner, or formally excluded and disclosed. A
    record with no disposition is an undisclosed omission from the total, and a
    growing queue is a shrinking reported figure.
    """
    stale, unowned = [], []
    for record in quarantine:
        age = (today - record["quarantined_on"]).days
        if record.get("disposition") is None:
            if age > max_age_days:
                stale.append(record["record_id"])
            if record.get("owner") is None:
                unowned.append(record["record_id"])

    result = {
        "queue_depth": len(quarantine),
        "unresolved": sum(1 for r in quarantine if r.get("disposition") is None),
        "stale": len(stale), "unowned": len(unowned),
        "oldest_days": max(((today - r["quarantined_on"]).days for r in quarantine),
                           default=0),
    }
    if stale or unowned:
        log.error("gate.quarantine_unresolved", **result,
                  example_stale=stale[:3], example_unowned=unowned[:3])
    return result


def audit_exceptions(exceptions: list[dict], today: date) -> dict:
    """A documented exception is a control weakening with an expiry date.

    One without an expiry is permanent by accident, which is how a temporary
    workaround becomes the reason a figure could not be substantiated.
    """
    expired = [e for e in exceptions
               if e.get("expires_on") and e["expires_on"] < today]
    perpetual = [e for e in exceptions if not e.get("expires_on")]

    result = {"active": len(exceptions), "expired_still_active": len(expired),
              "no_expiry": len(perpetual)}
    if expired or perpetual:
        log.error("gate.exceptions_unbounded", **result,
                  example=[e["exception_id"] for e in (expired + perpetual)[:3]])
    return result

The never_fired_in_two_years verdict is the one teams resist and should not. A check that has never fired is describing either something that genuinely cannot happen — in which case it is documentation rather than a control, and its cost is misleading confidence — or something it is failing to detect. Both readings warrant a deliberate test: feed the check a record you know should fail and confirm it does.

Where the suite runs against where the defects are introduced A four-stage pipeline with two overlays. The first overlay shows a validation suite attached only at ingestion, covering the first stage. The second overlay shows where defects are actually introduced, with a small share at ingestion and the majority spread across harmonisation, factor application and aggregation — a wrong reprojection, a stale factor version, a partial write. The uncovered region spanning the last three stages is highlighted. A panel notes that a suite attached at ingestion tests the source rather than the pipeline, and that the defects most damaging to a carbon figure are all introduced downstream of it. A suite at ingestion tests the source, not the pipeline Ingestion suite attached here Harmonisation wrong reprojection Factor application stale factor version Aggregation partial write covered uncovered — and where the damage happens Coverage decays without anyone changing a check. New stages are added downstream of an existing suite far more often than suites are extended to cover them.

Deterministic Transformation Logic

The controls that close the catalogue are mostly structural rather than clever. The gate runner below makes coercion impossible, records coverage per check, and refuses to report success when any blocking check was skipped.

from dataclasses import dataclass, asdict, field

import structlog

log = structlog.get_logger()


class GateSkipped(RuntimeError):
    """A blocking check did not run. Never a warning."""


@dataclass
class CheckOutcome:
    check_id: str
    blocking: bool
    ran: bool
    rows_evaluated: int
    rows_failed: int
    observed_examples: list = field(default_factory=list)
    skip_reason: str | None = None


def run_gate(records, checks, suite_version: str, run_id: str) -> dict:
    """Run every check on every row, or fail. No sampling, no skipping.

    Three properties close most of the catalogue:
      * a check either runs on the full scope or the run fails,
      * a failure quarantines rather than adjusts,
      * observed values are captured, because 'check failed' is not evidence.
    """
    outcomes: list[CheckOutcome] = []
    quarantined: list[dict] = []
    passed: list[dict] = []

    for record in records:
        record_failures = []
        for check in checks:
            if not check.applies_to(record):
                continue
            ok, observed = check.evaluate(record)
            if not ok:
                record_failures.append((check, observed))

        blocking_failures = [(c, o) for c, o in record_failures if c.blocking]
        if blocking_failures:
            quarantined.append({
                "record_id": record["record_id"],
                "failed_checks": [c.check_id for c, _ in blocking_failures],
                # Observed values, not just the check name — 'expectation failed'
                # is not something a reviewer can act on.
                "observed": {c.check_id: o for c, o in blocking_failures},
                "disposition": None, "owner": None,
            })
        else:
            if record_failures:
                record = {**record,
                          "advisory_flags": [c.check_id for c, _ in record_failures]}
            passed.append(record)

    for check in checks:
        scope = sum(1 for r in records if check.applies_to(r))
        evaluated = scope                     # no sampling: scope == evaluated
        failed = sum(1 for q in quarantined if check.check_id in q["failed_checks"])
        outcomes.append(CheckOutcome(
            check_id=check.check_id, blocking=check.blocking, ran=True,
            rows_evaluated=evaluated, rows_failed=failed,
            observed_examples=[q["observed"][check.check_id]
                               for q in quarantined
                               if check.check_id in q["observed"]][:3],
        ))

    skipped_blocking = [o.check_id for o in outcomes if o.blocking and not o.ran]
    if skipped_blocking:
        raise GateSkipped(f"blocking checks did not run: {skipped_blocking}")

    result = {
        "run_id": run_id, "suite_version": suite_version,
        "records_in": len(records), "records_passed": len(passed),
        "records_quarantined": len(quarantined),
        "checks": [asdict(o) for o in outcomes],
    }
    log.info("gate.complete", run_id=run_id, suite_version=suite_version,
             passed=len(passed), quarantined=len(quarantined),
             checks_run=len(outcomes))
    return {"passed": passed, "quarantined": quarantined, "report": result}


def assert_response_health(quarantine_audit: dict, exception_audit: dict,
                           max_queue: int = 500) -> None:
    """The response path is part of the control. Fail the period if it has rotted."""
    problems = []
    if quarantine_audit["stale"] > 0:
        problems.append(f"stale_quarantine:{quarantine_audit['stale']}")
    if quarantine_audit["unowned"] > 0:
        problems.append(f"unowned_quarantine:{quarantine_audit['unowned']}")
    if quarantine_audit["queue_depth"] > max_queue:
        problems.append(f"queue_depth:{quarantine_audit['queue_depth']}")
    if exception_audit["expired_still_active"] or exception_audit["no_expiry"]:
        problems.append("unbounded_exceptions")

    if problems:
        log.error("gate.response_path_degraded", problems=problems)
        raise RuntimeError(f"validation response path degraded: {problems}")

Two choices in that code do most of the work. There is no sampling parameter — the runner cannot be configured to check a subset, so the sampling failure mode is unreachable rather than merely discouraged. And response health is asserted as part of the period, which turns a growing quarantine queue and an expired exception from operational debt into something that fails the run, at the point where it is still cheap to fix.

Compliance Gating & Audit Trail Generation

The catalogue maps onto what a verifier asks about controls, and four artefacts answer it.

The suite version and its change history. A check-set change alters what “passed” means, so it is a control change requiring review, and the version must be recorded on every run. A figure produced under suite version 4 is not comparable to one produced under version 3 without knowing what changed.

Per-check coverage and fire counts. These are what distinguish a live control from a nominal one. A check that ran on every row and has fired eleven times this year is evidence; the same check with no coverage record is an assertion. Include the never-fired checks explicitly rather than omitting them.

The quarantine register with dispositions. Every record must exit through one of three doors — corrected, excepted with a named owner, or formally excluded and disclosed — and the register is where a verifier checks that the omissions in a reported total were deliberate and disclosed rather than accidental.

The exception log with expiries. An exception is a documented weakening of a control, and one without an expiry is a permanent weakening nobody decided on. Route all four into the evidence stream described under MRV pipeline observability and failure modes, keyed on the run identifier so they join to the figures they gated.

Fire rate against action rate, and the quadrant where a check is already ignored A scatter of validation checks with fire rate per period on the horizontal axis and the fraction of fires that led to a recorded action on the vertical axis. Checks in the upper left fire rarely and are always acted on, which is the healthy blocking pattern. Checks in the upper right fire often and are acted on, which usually means a genuine upstream problem worth fixing at source. Checks in the lower right fire often and are almost never acted on, marked as the fatigue quadrant where a check is being ignored whether or not anyone has muted it. Checks in the lower left fire rarely and were not acted on when they did, which points at an unclear runbook. A panel notes that the lower-right quadrant is measurable long before a channel is formally muted. Fatigue is measurable before anyone mutes anything Fire rate against the fraction of fires that produced a recorded action. fatigue quadrant 100% 50% 0 rare frequent action rate healthy blocking fix the source unclear runbook already ignored demote or delete

Production Integration

  1. Remove every coercion path from the gate layer, so a check can only pass or quarantine.
  2. Attach the suite at every stage boundary, not only at ingestion, and add coverage of a new stage to that stage’s definition of done.
  3. Forbid sampling in the runner rather than configuring it off, so the option cannot be re-enabled during an incident and forgotten.
  4. Fail the run when a blocking check did not execute, treating a skip exactly as severely as a failure.
  5. Audit the gates quarterly for coverage, fire rate, and dormancy, and deliberately test any check that has never fired.
  6. Assert response health as part of the period close — stale quarantine, unowned records, and unbounded exceptions all fail the period.

The cheapest high-value addition for a team retrofitting this is the coverage record. Knowing which checks ran on what fraction of which stages, per run, exposes most of the coverage group immediately and costs one counter per check.

Frequently Asked Questions

Is a check that has never fired a problem?

It is a question that deserves an answer rather than an assumption. A check may never have fired because the defect it guards against genuinely cannot occur in your pipeline — in which case it is documentation and its value is misleading confidence — or because it is miscalibrated and silently passing everything. The distinguishing test is cheap: construct a record you know should fail and confirm the check catches it. Do this once for every never-fired blocking check, and record the result as evidence the control is live.

How do I stop a validation suite from decaying as the pipeline grows?

Make suite coverage part of the definition of done for any new stage, and audit coverage on a schedule rather than trusting it. Decay happens because adding a stage is a visible task while extending the suite to cover it is not, so the fix is to make the second part of the first. A quarterly audit that reports checks-per-stage and rows-evaluated-versus-in-scope catches whatever slips through.

What is wrong with sampling validation for performance?

The sample is not where the defect is. Data defects in carbon pipelines cluster — one bad source file, one tile, one supplier — so a uniform sample of five per cent has a good chance of missing a cluster entirely while reporting a clean result. Where validation genuinely costs too much, the answer is to make the check cheaper or move it to a stage boundary rather than to look at fewer rows, because a partial check produces a confident statement about data it never saw.

How long should a record sit in quarantine?

Long enough to correct at source and no longer — typically a few weeks, with an explicit escalation when it ages past that. What matters more than the number is that every record exits through a recorded door. A record still in quarantine at period close is an omission from the reported total, and it must appear in the completeness statement whether or not anyone has looked at it.

Should exceptions ever be permanent?

No. Some will be long-lived — a data source that genuinely cannot supply a field, a legacy system that will be replaced eventually — but even those should carry an expiry that forces a periodic re-decision. An exception without an expiry stops being a decision and becomes a property of the system, and the reason it was granted is usually forgotten within a year, which is exactly when a verifier asks about it.

How should alert fatigue be measured before it causes damage?

By tracking the fire rate and the action rate per check, and comparing them. A check firing weekly with no recorded action is already being ignored regardless of whether anyone has muted the channel, and that gap is measurable long before the mute happens. Checks whose action rate is near zero should be demoted to trends or removed, since a channel containing anything unactionable degrades the credibility of everything else in it.

Does this catalogue apply to the geospatial invariants too?

Yes, and the response group especially. A CRS assertion or an area-drift invariant is subject to exactly the same decay — it can be skipped when a job is behind, its finding can land in an unread channel, and an exception granted for one odd dataset can outlive it. The cross-stage catalogue covers the geospatial checks themselves; the failure modes here are about the layer that runs them, and both sets are needed.

What is the right size for the blocking set?

Small enough that every member describes something never legitimate, which in practice means single digits per stage. The temptation is to promote checks to blocking as confidence in them grows, and the result is a gate that stops the pipeline for judgement calls at inconvenient hours. Keep blocking for absolutes — a missing unit, an absent coordinate reference system, a mass balance that does not close, a partition set that is incomplete — and leave everything with a legitimate exception to the advisory set where it can be trended.

A useful discipline when adding a check is to ask what the on-call response would be at three in the morning. If the answer is “look at it and probably override”, the check is advisory; if the answer is “stop, because this record cannot be interpreted”, it is blocking. Checks whose answer is unclear are the ones that later become the muted channel.