Spatial Pipeline EngineeringMRV Automation

Failure Mode Catalog for Distributed Tile Processing

A distributed raster job that crashes is a good outcome. Someone reads the traceback, fixes the cause, and reruns. The failures that cost carbon projects are the ones where every worker returns successfully, the output mosaic covers the full extent, the file sizes look plausible, and the numbers are wrong. This catalogue collects those, within async satellite tile processing with Dask in the satellite imagery processing for emissions tracking stack. It is the tile-processing counterpart to the cross-stage failure mode catalog for spatial MRV pipelines.

The organising property is that a tiled computation has no natural completeness check. A scalar sum either ran or did not. A mosaic of forty thousand tiles can be missing two hundred of them and look entirely normal on a map, because the missing tiles either carry the nodata value that the rest of the pipeline treats as legitimately absent, or carry a stale value from a previous run. Every entry below is a variant of that.

Ten distributed tile failures, grouped by where they originate A grid of ten failure modes in three groups. The partition group covers a chunk boundary cutting through a moving-window operation, a reprojection whose chunks do not align with the target grid, and a tile index that silently omits tiles at the extent edge. The execution group covers a worker killed for memory whose task is retried on data already partly written, a retry producing a different result because the operation is not deterministic, a lazy graph that never materialised the write, and a task that raised and was swallowed by a broad exception handler. The output group covers a partial write left behind after a failure, a nodata value indistinguishable from an unwritten tile, and a mosaic whose overviews were built from the previous run. A panel notes that none of the ten raise at the point of failure. Ten ways to finish successfully and be wrong The job exits zero. The mosaic covers the extent. The numbers are not the numbers. Partition — the split is wrong 1 · window spans a chunk edge 2 · chunks misaligned to target grid 3 · tile index omits edge tiles every task succeeds; the seams carry values that were never computed from full neighbourhoods Execution — it did not run 4 · OOM kill, retry over partial data 5 · retry gives a different answer 6 · lazy graph never materialised 7 · exception swallowed by except: the scheduler reports done because nothing told it otherwise Output — it looks complete 8 · partial write left in place 9 · nodata == never written 10 · overviews from the last run the map renders. the extent is covered. the histogram is plausible. nobody looks again.

Root Cause Analysis

The ten entries share three causes, and naming them shortens most diagnoses to a single question.

A chunk is an implementation detail that leaks into the result. Chunking is supposed to be invisible: the answer should not depend on how the array was split. It does depend on it whenever an operation has spatial extent — a focal filter, a segmentation, a connected-components labelling, a temporal gap-fill using neighbours. Dask will happily apply such an operation per chunk without overlap unless told otherwise, and the result is correct in the interior of every chunk and wrong along every boundary. On a map the errors form a visible grid; in a zonal statistic they are invisible and they bias the total.

Absence and zero are the same byte. A tile that was never written and a tile that was written with the nodata value are indistinguishable in the output. Because nodata is legitimately common in satellite products — cloud, scene edge, sensor gap — no downstream consumer can treat it as an error. This is the single property that makes silent tile loss undetectable without an explicit manifest, and it is why the completeness check has to be structural rather than statistical.

Retries assume determinism that the code does not provide. A retried task must produce the same result as the original for the job to be correct, and several common patterns break that: an unseeded random sample, a datetime.now() in an output path, a floating-point reduction whose order depends on arrival, an append to an existing file. When a retry produces a different result, the output depends on which workers happened to fail, which is not reproducible and cannot be audited.

The connective theme is that distributed frameworks provide task-level success and the pipeline needs data-level completeness, and nothing bridges the two automatically.

Diagnostic Pipeline / Pre-Flight Validation

The check that catches the largest share of this catalogue is a manifest comparison: enumerate what should exist before the job, enumerate what does exist after, and refuse to publish on any difference. It is unglamorous and it is the difference between a mosaic you can defend and one you hope is right.

from dataclasses import dataclass

import structlog

log = structlog.get_logger()


@dataclass(frozen=True)
class TileKey:
    """Identity of one output tile. Deterministic from the input grid alone."""
    grid_id: str
    col: int
    row: int

    def path(self, root: str, run_id: str) -> str:
        return f"{root}/{run_id}/{self.grid_id}/{self.col:04d}_{self.row:04d}.tif"


@dataclass(frozen=True)
class TileResult:
    key: TileKey
    bytes_written: int
    checksum: str
    valid_pixel_count: int
    nodata_pixel_count: int
    worker: str
    attempt: int


class IncompleteMosaicError(RuntimeError):
    """Raised when the produced set differs from the expected set."""


def expected_tiles(grid_id: str, n_cols: int, n_rows: int) -> set[TileKey]:
    """The full expected set, computed from the grid, not from the outputs.

    Deriving the expectation from what was produced is the mistake this
    function exists to prevent — it makes any loss self-consistent.
    """
    return {
        TileKey(grid_id, c, r)
        for c in range(n_cols)
        for r in range(n_rows)
    }


def assert_mosaic_complete(
    expected: set[TileKey], produced: list[TileResult]
) -> None:
    """Structural completeness. No statistics, no thresholds, no tolerance."""
    produced_keys = {t.key for t in produced}

    missing = expected - produced_keys
    unexpected = produced_keys - expected

    duplicates = [
        k for k in produced_keys
        if sum(1 for t in produced if t.key == k) > 1
    ]

    if missing or unexpected or duplicates:
        raise IncompleteMosaicError(
            f"expected {len(expected)} tiles, produced {len(produced_keys)}; "
            f"missing {len(missing)}, unexpected {len(unexpected)}, "
            f"duplicated {len(duplicates)}. "
            f"first missing: {sorted(missing)[:5] if missing else '—'}"
        )

    empty = [t for t in produced if t.valid_pixel_count == 0]
    if empty:
        # Not necessarily an error — a tile can be legitimately all-cloud —
        # but it must be counted and reported rather than absorbed.
        log.warning(
            "mosaic.all_nodata_tiles",
            count=len(empty),
            fraction=round(len(empty) / len(produced), 4),
            sample=[f"{t.key.col}_{t.key.row}" for t in empty[:5]],
        )

    retried = [t for t in produced if t.attempt > 1]
    if retried:
        log.info(
            "mosaic.retried_tiles",
            count=len(retried),
            note="verify the operation is deterministic before trusting these",
        )

    log.info(
        "mosaic.complete",
        tiles=len(produced),
        valid_pixels=sum(t.valid_pixel_count for t in produced),
        nodata_pixels=sum(t.nodata_pixel_count for t in produced),
    )


def assert_overlap_sufficient(
    window_radius_px: int, chunk_overlap_px: int, operation: str
) -> None:
    """Chunk overlap must exceed the operation's spatial reach.

    Applies to focal filters, morphological operations, segmentation, and
    any temporal fill that reads spatial neighbours. The check is cheap and
    the failure it prevents is invisible.
    """
    if chunk_overlap_px < window_radius_px:
        raise ValueError(
            f"operation '{operation}' reaches {window_radius_px} px but chunks "
            f"overlap by only {chunk_overlap_px} px — every chunk boundary will "
            "carry values computed from a truncated neighbourhood. Use "
            "map_overlap with depth >= the window radius."
        )

The duplicate check in the middle catches a specific and nasty case: a retry that writes to a slightly different path, leaving both the failed partial and the successful complete tile in the output directory. The mosaic then contains two candidates for one position and whichever the reader picks first wins.

A focal operation across a chunk boundary, with and without overlap Two panels showing a three by three moving window applied near a chunk boundary. On the left, chunks are processed independently with no overlap: the window at a pixel adjacent to the boundary reads only the pixels inside its own chunk, treating the missing neighbours as absent, so the computed value is derived from a truncated neighbourhood. A visible seam of wrong values runs the length of every boundary. On the right, chunks are expanded by a halo equal to the window radius before processing and trimmed afterwards: every window reads its full neighbourhood, and the result is identical to what a single-machine computation would produce. A panel notes that on a forty thousand tile mosaic the seams amount to a substantial fraction of the pixels, and that they bias any zonal total computed from the result. The seam is not cosmetic — it biases every zonal total downstream window truncated at the boundary No overlap every task succeeds; a seam of wrong values runs along every chunk edge 40,000 tiles × 4 edges = a lot of pixels halo supplies the full neighbourhood map_overlap, depth ≥ radius chunks expand, compute, then trim result matches single-machine exactly costs memory, not correctness

Deterministic Transformation Logic

Making a tiled job safe to retry is largely a matter of two disciplines: every task writes to a path derived only from its inputs, and every write is atomic. Together they make a retry indistinguishable from a first attempt.

import hashlib
import json
import os
from dataclasses import dataclass
from pathlib import Path


@dataclass(frozen=True)
class TaskSpec:
    """Everything that determines a tile's content, and nothing else.

    Notably absent: timestamps, hostnames, attempt numbers, and run ids used
    for anything other than directory placement. If any of those enter the
    computation, a retry can produce a different tile and the job stops being
    reproducible.
    """
    key: TileKey
    source_uris: tuple[str, ...]
    algorithm: str
    algorithm_version: str
    parameters: tuple[tuple[str, str], ...]

    def fingerprint(self) -> str:
        payload = json.dumps(
            {
                "key": [self.key.grid_id, self.key.col, self.key.row],
                "sources": sorted(self.source_uris),
                "algorithm": self.algorithm,
                "version": self.algorithm_version,
                "parameters": sorted(self.parameters),
            },
            sort_keys=True,
            separators=(",", ":"),
        ).encode()
        return hashlib.sha256(payload).hexdigest()[:16]


def write_tile_atomically(dest: Path, payload: bytes) -> None:
    """Write to a temporary name in the same directory, then rename.

    Rename within a filesystem is atomic, so a reader either sees the whole
    tile or no tile at all — never a truncated one. The temporary name must
    share the destination directory, or the rename becomes a copy and loses
    atomicity. On object storage the equivalent is a single-part put or a
    completed multipart upload; never an appended stream.
    """
    dest.parent.mkdir(parents=True, exist_ok=True)
    tmp = dest.with_name(f".{dest.name}.partial")
    try:
        with open(tmp, "wb") as fh:
            fh.write(payload)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp, dest)
    except BaseException:
        tmp.unlink(missing_ok=True)
        raise


def process_tile(spec: TaskSpec, root: str, run_id: str) -> TileResult:
    """Idempotent tile production: skip if the fingerprint already exists.

    The fingerprint in the filename is what makes the skip safe. A tile whose
    inputs or parameters changed produces a different fingerprint and is
    therefore recomputed, while an identical rerun costs a stat() call.
    """
    fp = spec.fingerprint()
    dest = Path(f"{root}/{run_id}/{spec.key.grid_id}/"
                f"{spec.key.col:04d}_{spec.key.row:04d}.{fp}.tif")

    if dest.exists():
        log.info("tile.skipped_existing", tile=str(dest.name), fingerprint=fp)
        return _describe(dest, spec, attempt=0)

    payload, stats = _compute(spec)
    write_tile_atomically(dest, payload)

    return TileResult(
        key=spec.key,
        bytes_written=len(payload),
        checksum=hashlib.sha256(payload).hexdigest(),
        valid_pixel_count=stats["valid"],
        nodata_pixel_count=stats["nodata"],
        worker=os.environ.get("DASK_WORKER_NAME", "local"),
        attempt=1,
    )

Putting the fingerprint in the filename does double duty. It makes reruns cheap, and it makes a parameter change visible in the filesystem — a directory containing two fingerprints for the same tile position is the immediate, obvious signal that something about the computation changed mid-run, which is otherwise one of the hardest states to detect.

Compliance Gating & Audit Trail Generation

The gate before publication is the manifest comparison, and it should block rather than warn. Beyond it, four records make a tiled output auditable.

The expected tile set and its derivation. A verifier needs to see that completeness was checked against the grid definition rather than against whatever was produced.

Per-tile checksums and fingerprints. These allow any single tile to be re-derived and compared, which is the practical form of reproducibility for a dataset too large to regenerate wholesale.

Retry counts and their outcomes. A run with three hundred retries is not necessarily wrong, but it is a run whose determinism claim is load-bearing, and the count is what prompts anyone to check it.

The chunking and overlap parameters used. These are part of the computation’s definition even though they feel like infrastructure, because an operation with spatial reach produces a different answer under different chunking. Recording them turns “why does the 2027 mosaic differ slightly from the 2026 one?” into a diff rather than an investigation.

Production Integration

Most of this catalogue is prevented by three habits rather than by tooling. Derive the expected output set from the input grid before the job starts and store it. Make every write atomic and every path a pure function of the task’s inputs. Refuse to publish on any manifest mismatch, without a threshold, because a tolerance for missing tiles is a tolerance for a wrong answer of unknown size.

The scaling patterns themselves are covered in scaling async satellite processing with Dask geospatial, and the instrumentation that makes the failures above visible in flight rather than at publication is in instrumenting MRV pipelines with OpenTelemetry and structlog. The two combine usefully: a span per tile with the fingerprint as an attribute turns the retry-determinism question into a query.

Where the completeness gate sits, and what it stops A flow from left to right. A grid definition produces an expected tile manifest before any work starts. Workers produce tiles in parallel, each writing atomically to a fingerprinted path. A completeness gate compares produced against expected and blocks on any difference — missing, unexpected, or duplicated. Only after the gate passes does the mosaic get assembled, overviews built, and the output promoted to the certified zone. A red branch from the gate shows the blocked path, annotated that a blocked run leaves the previous certified output in place rather than replacing it with a partial one, so a consumer reading the certified zone never sees an incomplete mosaic. Expect first, produce second, compare third — publish only on an exact match Expected manifest from the grid definition, before any work starts Workers atomic writes to fingerprinted paths Completeness gate missing / unexpected / duplicated → block Publish mosaic, overviews, promote to certified Blocked: the previous certified mosaic stays in place. Consumers never see a partial mosaic — which is why the gate precedes promotion. on mismatch

Frequently Asked Questions

How large should chunks be?

Large enough that per-task overhead is negligible and small enough that several fit in a worker’s memory alongside their halos — in practice a few hundred megabytes of uncompressed array per chunk suits most raster work. The consideration people miss is the halo: a chunk with a fifty-pixel overlap on each side carries substantially more data than its nominal size, and a chunking that fits comfortably without overlap can trigger memory pressure once the halo is added. Size the chunk for the halo it will actually carry.

Is it safe to let the framework retry failed tasks automatically?

Only if the task is genuinely deterministic and its write is atomic. With those two properties, automatic retry is exactly right and removes a large class of transient failures. Without them, retry converts a transient failure into a silent corruption, because the second attempt may leave different bytes on disk from the first and nothing compares them. The honest position is that automatic retry is a property you earn by making tasks pure, not a setting you enable.

Why does a mosaic sometimes differ between runs with identical inputs?

Usually one of three things: a reduction whose floating-point order varies with task arrival, an operation reading a mutable external resource such as a “latest” pointer, or an unseeded sampling step. Floating-point non-associativity is the subtlest — summing the same values in a different order gives results that differ in the last bits, which is harmless for most purposes and fatal for a checksum comparison. Where bitwise reproducibility is needed, sum in a fixed order or use a compensated summation.

Should nodata and never-written be distinguished in the output format?

Ideally yes, and there are two workable approaches. A separate validity mask band records whether each pixel was computed, independently of what value it holds. Or a distinct sentinel is used for never-written, though this requires every consumer to know about it and is easily lost in a format conversion. Where neither is practical, the tile manifest carries the burden: if the manifest is complete and each tile records its valid pixel count, absence at pixel level can be attributed correctly.

What is the right response to a worker being killed for memory?

Investigate before increasing the limit, because an out-of-memory kill in a tiled job usually means a task is loading more than its chunk. Common causes are a compute() inside a task pulling the whole array, an unexpected broadcast, or a halo larger than intended. Raising the memory limit makes the symptom go away and leaves a job that will fail again at the next scale increase. Where the memory use is genuinely necessary, reduce the chunk size rather than raising the limit — it scales further.

How should partial writes from a killed worker be cleaned up?

They should not need cleaning up, because the temporary-then-rename pattern leaves only a dot-prefixed partial file that no reader will match. That said, a periodic sweep removing stale partials is worth having, since they accumulate and consume storage. What must never happen is a cleanup that removes files matching the real output pattern, since a concurrent run’s legitimate tiles look identical to a stale one from the outside.

Does this catalogue apply to Spark and Ray as well as Dask?

Almost entirely. The specific APIs differ — map_overlap has counterparts elsewhere, and the atomic-write pattern is universal — but the three root causes are properties of distributing spatial computation rather than of any framework. The one framework-specific entry is the lazy-graph failure, which affects Dask and Spark in the same way and does not arise in eagerly evaluated systems. Everything about manifests, determinism, and atomicity transfers unchanged.

Does a completeness gate slow the pipeline down noticeably?

Not measurably, because it operates on metadata rather than on pixels. Comparing two sets of a few tens of thousands of keys is microseconds of work, and the per-tile records it consumes are produced by the tasks anyway. The cost that people actually feel is the blocked run: a gate that refuses to publish forces someone to investigate a mismatch that a permissive pipeline would have shipped. That is the gate working, and the time it takes is time that would otherwise have been spent months later reconstructing why a total moved.

The one real cost is at very large tile counts, where holding the produced set in memory on the coordinating process becomes awkward. At that scale, compare sorted key streams rather than sets, or partition the comparison by grid row — both preserve the exactness of the check, which is the property that must not be traded away.

Should the expected manifest include tiles that fall entirely outside the area of interest?

Yes, and mark them expected-empty rather than excluding them. Excluding them means the expected set depends on a geometry intersection, which is itself a computation that can go wrong, and a bug there quietly shrinks the expectation to match whatever was produced. Including every tile in the grid rectangle and recording which are expected to contain no valid pixels keeps the expectation derived from something structural — the grid — while still letting the gate distinguish a legitimately empty tile from a lost one.