Spatial Pipeline EngineeringMRV Automation

Failure Mode Catalog for Threshold Drift in Carbon Baselines

A threshold is the most innocuous-looking number in a carbon pipeline and the one with the most leverage. A canopy cover cut-off decides what counts as forest, which decides the baseline area, which multiplies through every subsequent figure. Because the number itself rarely changes, threshold problems are almost never detected as changes to a threshold — they are detected, if at all, as an unexplained movement in a total. This catalogue collects those cases, within threshold tuning for carbon stock baselines in the spatial modeling and carbon stock validation stack.

The organising observation is that a threshold is a boundary drawn on a distribution, and it is the distribution that moves. Nobody edits the number; the imagery changes collection, the index gets recalibrated, a new sensor joins the input mix, the atmospheric correction improves — and the same numeric cut-off now falls at a different point in the data. The threshold has drifted without being touched, which is why version control on the configuration file does not detect it.

Ten threshold drift modes, grouped by what moved A grid of ten failure modes in three groups. The input group covers a collection or processing baseline change shifting the index distribution, a new sensor joining the mix with a different band response, an atmospheric correction improvement altering reflectance systematically, and a resampling change altering the pixel support. The definition group covers a threshold fitted once on one biome then applied everywhere, a cut-off calibrated against a reference dataset that was itself later revised, and a minimum mapping unit changed independently of the cover threshold. The governance group covers a threshold retuned until the credited area reached a target, a threshold changed without restating prior periods, and a threshold whose original justification nobody can locate. A panel notes that in nine of the ten the numeric threshold never changed. Ten ways a threshold moves without being edited In nine of the ten, the number in the config file is unchanged. Input — the distribution moved 1 · collection / baseline change 2 · new sensor in the mix 3 · atmospheric correction improved 4 · resampling changed the support the cut-off is where it always was; the data slid underneath it Definition — it never fitted 5 · fitted on one biome, used on all 6 · reference data later revised 7 · MMU changed independently correct where it was fitted and wrong everywhere else, from the first day it was applied Governance — it was moved 8 · retuned until the area fitted 9 · changed without restating priors 10 · original justification lost the only group a verifier can see directly — and the one that ends a validation

Root Cause Analysis

The ten entries reduce to three causes, and separating them decides who has to fix what.

A threshold is meaningful only relative to the distribution it was fitted on. A canopy cover cut-off of thirty percent expresses a judgement about where forest begins, but the pipeline implements it as a numeric comparison against a modelled cover value, and that model’s output distribution depends on the imagery, the algorithm, and the processing chain. Change any of those and the same numeric comparison implements a different judgement. The threshold that survives is one recorded alongside the distribution it was calibrated against, so a shift in the distribution is detectable as a shift.

Thresholds interact, and are usually tuned in isolation. Canopy cover, minimum mapping unit, and minimum width jointly define forest, and moving any one changes what the others exclude. Raising the cover threshold shrinks patches, which pushes more of them under the minimum mapping unit, which removes more area than the cover change alone accounts for. Teams tune one at a time, measure the effect of each, and are then surprised by the combined result — which is not the sum of the parts.

Tuning against the outcome is nearly irresistible and completely fatal. Every threshold has a range of defensible values, and within that range the credited area varies substantially. A team that tries several values and selects the one producing the expected area has performed a legitimate-looking sensitivity analysis and arrived at a number chosen by its result. This is the failure that ends validations, and it is detectable in the record: a threshold whose selection log shows the credited area computed before the choice was made is very hard to defend afterwards.

The thread running through all three is that the threshold’s value is not the artefact — the justification is. A number without a recorded basis cannot be assessed, defended, or safely changed.

Diagnostic Pipeline / Pre-Flight Validation

The most effective detector is a distribution monitor: record the distribution of the thresholded variable each period, and alert when it moves relative to the period the threshold was calibrated on. This catches every entry in the input group before it reaches a reported figure.

from dataclasses import dataclass
from datetime import date

import numpy as np
import structlog

log = structlog.get_logger()


@dataclass(frozen=True)
class ThresholdSpec:
    """A threshold plus everything needed to tell whether it still applies."""
    threshold_id: str
    variable: str
    value: float
    calibrated_on: date
    calibration_source: str          # imagery collection + version
    calibration_biome: str
    reference_dataset: str
    reference_version: str
    justification: str
    # The calibration-period distribution, as percentiles. This is what
    # makes drift detectable — the number alone cannot move, so it cannot
    # be monitored; the distribution beneath it can.
    calib_percentiles: tuple[float, ...]   # p05..p95 in steps of 5


@dataclass(frozen=True)
class DriftReport:
    threshold_id: str
    period: str
    percentile_at_threshold_then: float
    percentile_at_threshold_now: float
    shift_pp: float
    area_change_pct: float
    verdict: str                     # stable | review | blocked


DRIFT_REVIEW_PP = 2.0
DRIFT_BLOCK_PP = 5.0


def percentile_of(value: float, percentiles: tuple[float, ...]) -> float:
    """Where a fixed value sits in a distribution given as p05..p95."""
    levels = np.arange(5, 100, 5)
    return float(np.interp(value, np.array(percentiles), levels))


def detect_drift(
    spec: ThresholdSpec, current_percentiles: tuple[float, ...],
    period: str, area_change_pct: float,
) -> DriftReport:
    """Has the distribution moved under a fixed threshold?

    The question is not whether the threshold value changed — it has not.
    It is whether the same value now separates a different share of the
    landscape, which is the operative definition of drift.
    """
    then = percentile_of(spec.value, spec.calib_percentiles)
    now = percentile_of(spec.value, current_percentiles)
    shift = abs(now - then)

    if shift >= DRIFT_BLOCK_PP:
        verdict = "blocked"
        log.error(
            "threshold.drift_blocked",
            threshold=spec.threshold_id, period=period,
            percentile_then=round(then, 1), percentile_now=round(now, 1),
            shift_pp=round(shift, 1),
            note="the same cut-off now classifies a materially different "
                 "share of the landscape; recalibrate or restate, do not "
                 "publish against a threshold that has silently moved",
        )
    elif shift >= DRIFT_REVIEW_PP:
        verdict = "review"
        log.warning(
            "threshold.drift_review", threshold=spec.threshold_id,
            period=period, shift_pp=round(shift, 1),
        )
    else:
        verdict = "stable"

    return DriftReport(
        spec.threshold_id, period, round(then, 2), round(now, 2),
        round(shift, 2), area_change_pct, verdict,
    )


def assert_applicable(spec: ThresholdSpec, biome: str, source: str) -> None:
    """Refuse to apply a threshold outside the conditions it was fitted for."""
    if spec.calibration_biome != biome:
        raise ValueError(
            f"threshold {spec.threshold_id} was calibrated on "
            f"'{spec.calibration_biome}' and is being applied to '{biome}'. "
            "Fit a threshold per biome or state the transfer explicitly — a "
            "cover cut-off from closed tropical forest classifies open "
            "woodland almost arbitrarily."
        )
    if spec.calibration_source != source:
        log.warning(
            "threshold.source_mismatch",
            threshold=spec.threshold_id,
            calibrated_on=spec.calibration_source, applied_to=source,
            note="run drift detection before trusting this period",
        )

The percentile framing is what makes this monitorable. A threshold of 0.30 is unchanged every period by construction; the share of the landscape it separates is a number that moves, and moving numbers can be alerted on.

A fixed threshold over two distributions after a collection change Two overlapping distributions of a canopy cover index across a landscape. The calibration-period distribution is drawn solid, with a vertical threshold line at the fitted value falling at the twenty-eighth percentile, so twenty-eight percent of the landscape falls below the cut-off and is classified as non-forest. The current-period distribution, produced after a collection change that shifted the index slightly upward, is drawn dashed and displaced to the right. The same vertical threshold line now falls at the twenty-first percentile of the new distribution, so only twenty-one percent falls below it and seven percent of the landscape has changed class without anything changing on the ground. A panel notes that the threshold value is identical in both cases, so a diff of the configuration shows nothing. The line did not move. The landscape underneath it did. threshold = 0.30, unchanged modelled canopy cover → density Calibration period 28% of the landscape falls below the cut-off After a collection change 21% falls below it 7% of the landscape changed class for free a config diff shows absolutely nothing

Deterministic Transformation Logic

Once a threshold is treated as a versioned object rather than a constant, the pipeline can apply the right one to each period and produce a comparable series. Two rules make that work: a threshold change creates a new version rather than editing the old one, and every classified output records which version produced it.

from dataclasses import dataclass, replace
from datetime import date


@dataclass(frozen=True)
class ThresholdVersion:
    spec: ThresholdSpec
    version: int
    effective_from: date
    effective_to: date | None
    supersedes: int | None
    change_reason: str


class ThresholdRegistry:
    """Append-only store of threshold versions with as-of resolution.

    Editing a threshold in place is the single change that makes a carbon
    time series non-comparable and non-auditable at once, so this class has
    no update method at all.
    """

    def __init__(self) -> None:
        self._versions: dict[str, list[ThresholdVersion]] = {}

    def add(self, tv: ThresholdVersion) -> None:
        chain = self._versions.setdefault(tv.spec.threshold_id, [])
        if chain and chain[-1].effective_to is None:
            chain[-1] = replace(chain[-1], effective_to=tv.effective_from)
        if not tv.change_reason.strip():
            raise ValueError(
                "a threshold version with no stated reason is a threshold "
                "nobody can defend later; state why it changed"
            )
        chain.append(tv)

    def resolve(self, threshold_id: str, on: date) -> ThresholdVersion:
        for tv in self._versions.get(threshold_id, []):
            if tv.effective_from <= on and (
                tv.effective_to is None or on < tv.effective_to
            ):
                return tv
        raise KeyError(
            f"no version of threshold '{threshold_id}' is effective on {on}"
        )

    def restatement_impact(
        self, threshold_id: str, from_version: int, to_version: int
    ) -> str:
        """What changing versions means for periods already reported.

        Called before a threshold change is committed, not after. A change
        that would alter a previously reported area is a restatement, and
        deciding that consciously beats discovering it in a verification.
        """
        chain = self._versions[threshold_id]
        old = next(v for v in chain if v.version == from_version)
        new = next(v for v in chain if v.version == to_version)
        return (
            f"threshold {threshold_id}: {old.spec.value} → {new.spec.value} "
            f"effective {new.effective_from}. Periods before that date keep "
            f"version {from_version}; recomputing them under {to_version} is "
            "a restatement and must be reported as one."
        )


def classify_with_provenance(
    values: list[float], tv: ThresholdVersion
) -> list[tuple[bool, str]]:
    """Classify and stamp. The stamp is not optional overhead.

    Without it, a mixed-vintage dataset — some periods classified under one
    version, some under another — is indistinguishable from a consistent one,
    and any trend computed across the boundary is partly an artefact of the
    threshold change.
    """
    stamp = f"{tv.spec.threshold_id}@v{tv.version}"
    return [(v >= tv.spec.value, stamp) for v in values]

The refusal to accept an empty change reason is a small guard with an outsized return. Threshold changes are usually made under time pressure and their reasons are obvious at the moment they are made and irrecoverable eighteen months later, when a verifier asks why the forest definition tightened between two periods.

Compliance Gating & Audit Trail Generation

Four records make thresholds defensible, and they are all cheap relative to the consequence of not having them.

The threshold’s calibration basis: the reference dataset and version, the biome, the imagery collection, and the distribution it was fitted against. This is the artefact that answers “why this value” without recourse to memory.

The drift report per period. A stable verdict every period is a strong statement; the absence of any drift monitoring is what invites the question of whether the threshold still means anything.

The version stamp on every classified output. This is what allows a multi-year series to be checked for consistency mechanically, and what makes a mixed-vintage comparison visible rather than silent.

The selection record, showing what was considered and in what order. This is the defence against the accusation of outcome tuning, and it works only if the record shows the sensitivity analysis was run and the choice justified on grounds other than the resulting area. A record that shows credited area computed for each candidate before the selection is worse than no record at all.

Production Integration

The registry belongs alongside the emission factor tables rather than in a pipeline configuration file, because the two have identical requirements: append-only versions, validity intervals, mandatory as-of resolution, and a refusal to interpolate. A project that has implemented versioning emission factor databases for reproducible MRV already has the machinery and needs only to register thresholds in it.

The drift monitor should run on every period regardless of whether anything is expected to have changed, since the whole point is that changes arrive from upstream without announcement. Where it fires, the response is a choice between recalibrating the threshold — which creates a version and a restatement question — and rejecting the upstream change, which is occasionally the right answer when a collection update is known to be problematic. The tuning mechanics themselves are covered in tuning canopy cover thresholds for forest baselines, and the same precision-omission trade appears in alerting, described in reducing false positive deforestation alerts.

Three thresholds interacting, and why tuning them one at a time misleads A diagram showing three interacting definitions of forest applied in sequence to the same landscape. A canopy cover cut-off removes the sparsest areas. A minimum mapping unit then removes patches below a size, and because the cover cut-off has already fragmented some patches, it removes more area than it would have on the unfiltered landscape. A minimum width rule then removes narrow strips, which the previous two steps have also made more numerous. Three bars show the area removed by each rule measured in isolation, and a fourth bar shows the area removed when all three are applied together, which is substantially larger than their sum would suggest. A panel notes that the interaction is why a sensitivity analysis must vary the rules jointly rather than one at a time. The three rules are not independent — and the combined effect exceeds the sum canopy cover ≥ 30% patch ≥ 0.5 ha width ≥ 20 m all three together area removed from the forest baseline 4.1% measured alone 2.1% measured alone 1.4% measured alone sum of the three: 7.6% actual: 10.7% A cover cut-off fragments patches, which pushes more of them under the size and width rules. Vary the three jointly, or the analysis understates by a third.

Frequently Asked Questions

How large a distribution shift should block publication?

The percentile framing gives a natural scale, and a shift of about five percentage points in where the threshold sits is a reasonable blocking level for a forest cover definition — at that magnitude the classified area has moved enough to be visible in a reported total. Two percentage points is a sensible review level. Those figures are starting points rather than standards; the right number for a given project follows from how much area movement its methodology tolerates before a restatement is required.

Is recalibrating a threshold after a collection change a restatement?

Recalibrating so that the new imagery classifies the landscape as the old imagery did is a continuity measure and is generally the right response. It becomes a restatement when prior periods are recomputed under the new value. The clean approach is to version the threshold with an effective date matching the collection change, leave prior periods on the prior version, and state in the monitoring report that the definition was held constant in substance while its numeric expression changed.

How should a threshold be chosen without tuning to the outcome?

Choose it against an independent reference — a set of interpreted points, a field-verified sample, a national forest definition — and select the value that best matches that reference, with the credited area computed only afterwards. The sequencing is the control. Where the resulting area is uncomfortable, the honest move is to note the discomfort and keep the value, since the alternative leaves a record showing the area was known before the choice was made.

Do these problems apply to continuous outputs, or only to classifications?

Mostly to classifications, because a threshold is what creates a class. Continuous outputs suffer a related but milder version: a distribution shift moves the values without creating a discontinuity, so the effect appears as a gradual trend rather than an abrupt area change. That is arguably worse, since a trend looks like a finding. The same drift monitor detects it, and comparing the current distribution against the calibration one is the check in both cases.

What if the calibration-period distribution was never recorded?

Reconstruct it if the imagery is still available, which it usually is — reprocessing a sample of the calibration-period scenes under the original collection gives a workable approximation. Where reconstruction is impossible, the honest position is that drift cannot be detected for that threshold and the next recalibration establishes a new baseline going forward. Recording that limitation is better than asserting stability that has not been checked.

Should thresholds ever differ between the baseline and monitoring periods?

No, and this is one of the few genuinely absolute rules here. The baseline and the monitoring period are compared to each other, and a definition that differs between them makes the comparison meaningless — any change measured is partly a change in the definition. Where a threshold must change mid-project because of an upstream change, both the baseline and the monitoring period should be reprocessed under the new version, and the resulting restatement reported.

How does the minimum mapping unit interact with pixel size?

Directly, and it is a common source of unnoticed drift. A minimum mapping unit expressed in pixels rather than in hectares changes meaning the moment the pixel size changes — a five-pixel minimum is 0.45 ha at thirty metres and 0.05 ha at ten. Adding Sentinel-2 to a Landsat-based pipeline therefore loosens the rule by an order of magnitude without anyone editing it. Express every spatial rule in ground units, and let the pipeline convert to pixels at the point of use.

Can a threshold be avoided altogether?

Sometimes, and where it can be it usually should be. Reporting a continuous carbon density surface and integrating it over the project area needs no forest definition at all, because nothing is being classified — every hectare contributes what it holds. Thresholds become unavoidable when the methodology’s unit of account is an area of forest rather than a quantity of carbon, which is the case for most deforestation-based crediting and for most regulatory land cover reporting.

Where a threshold is unavoidable, the next best mitigation is to report the sensitivity alongside the figure: the credited area at the chosen value and at the two neighbouring defensible values. That converts a single number resting on a judgement into a range with the judgement visible, and it removes most of the force of a challenge to the specific value, because the answer to “why 30 rather than 25?” is already on the page.