Spatial Pipeline EngineeringMRV Automation

MRV Data Lineage & Provenance Tracking

MRV data lineage and provenance tracking is the append-only evidence layer that binds every satellite pixel, ground plot, and modeled carbon-stock estimate to its source, transformation logic, and spatial reference system so that a reported tonne can be reconstructed byte-for-byte by a third-party verifier — and it is the connective tissue of the MRV Architecture & Carbon Accounting Fundamentals stack. It is not a compliance checkbox bolted on after the numbers are produced. When a pipeline reduces terabytes of multi-temporal imagery to a single CO₂-equivalent figure, lineage is the only mechanism that lets an auditor distinguish a defensible result from an unfalsifiable one.

This component sits directly downstream of geospatial CRS alignment, inheriting the coordinate-handling decisions made at ingestion, and directly upstream of carbon credit registry data integration, which cannot submit a figure it cannot trace. It also threads through GHG Protocol Scope 3 spatial mapping, where supply-chain attribution across fragmented geographies is meaningless unless each emission factor and boundary intersection carries its own recorded origin. This article focuses on the satellite-to-carbon-stock synchronization stage, where spatial drift, cloud-masking artifacts, and CRS misalignment most frequently sever the audit trail — and shows how deterministic provenance capture keeps that chain intact.

Provenance-stamped MRV synchronization: every branch converges into one append-only manifest Optical imagery flows into a cloud-mask and single-pass CRS-alignment step, then into a carbon-stock computation decision. A success branch yields a registry-ready carbon-stock raster; a failure branch routes to a fallback dataset logged with its failure context. Both branches converge into an append-only provenance manifest that records the SHA-256 checksum, parameter snapshot, and CRS for every node. INPUT Optical imagery Align & cloud mask single-pass reproject → canonical CRS Carbon stock computation success failure Carbon stock raster registry-ready GeoTIFF Fallback dataset logged with failure context APPEND-ONLY PROVENANCE MANIFEST one node per transformation SHA-256 · params · CRS · status

Role in the MRV Workflow

Provenance tracking is a cross-cutting concern rather than a single pipeline stage: it instruments the ingestion, spatial-normalization, factor-application, aggregation, and verification stages alike, attaching an immutable record to each artifact those stages emit. In the synchronization workflow examined here, the component wraps a tight sequence — ingest high-resolution optical imagery, apply atmospheric and cloud correction, align outputs to a canonical project CRS, compute a biomass or soil-carbon proxy, and serialize a registry-ready GeoTIFF alongside a metadata manifest. Each transformation is logged with input/output paths, parameter snapshots, the active spatial reference, and a content hash of the resulting artifact.

The upstream dependency is the canonical ingestion schema and the aligned geometry produced by CRS harmonization. If a raster arrives without an explicit, machine-readable datum tag, lineage capture has nothing trustworthy to record, so the discipline begins at the moment of acquisition rather than at export. The downstream consumers are unforgiving. A registry verifier reconstructing a credit volume must be able to walk backward from the certified tonnage to the exact tile, the exact cloud threshold, the exact allometric calibration, and the exact reprojection grid that produced it. When that walk-back is impossible, the figure is unverifiable regardless of whether it is numerically correct.

Crucially, the synchronization stage produces more than a carbon raster — it produces evidence. Modern pipelines treat lineage as a first-class output committed before the next stage reads anything, which is what makes a failed run at aggregation re-runnable from a stage-three artifact without re-ingesting raw telemetry. That contract — every stage commits an addressable, hashed artifact and a provenance node before the next stage begins — is what turns a pile of intermediate rasters into a queryable audit record that feeds enterprise event schemas such as tracking data lineage with OpenLineage for ESG audits.

Core Failure Modes

Three failure modes dominate production provenance capture in satellite-to-carbon synchronization. Each has a distinct root cause and a measurable impact on audit integrity.

  1. Silent CRS drift across iterative reprojection. When intermediate rasters are reprojected from previously reprojected derivatives rather than from the authoritative source, IEEE 754 precision loss and repeated resampling walk vertices by sub-pixel amounts that compound across monitoring cycles. The provenance failure is worse than the geometric one: if the lineage node records only the final CRS and not each warp in the chain, the drift is invisible to the auditor. A 30-meter cumulative shift on a 50-hectare parcel can misallocate several hectares across a project boundary while the manifest still reports a clean EPSG:4326 tag, silently invalidating the intersection logic that decides which pixels earn credit.

  2. Cloud-mask artifacts recorded without parameter snapshots. A cloud mask tuned to one scene’s illumination is wrong for the next, and if the masking threshold is applied but not captured in the lineage node, two runs over the same tile can produce materially different carbon proxies with no recorded explanation. Persistent haze or sensor saturation can suppress 10–40% of valid pixels; without the threshold, the band ratio, and the masked-pixel fraction logged per node, an auditor cannot reproduce the carbon figure and the run fails traceability on its face.

  3. Untracked fallback substitution. When primary computation fails — sensor degradation, corrupt tile boundaries, or unrecoverable cloud cover — pipelines that quietly swap in a fallback dataset destroy the chain of custody. The substituted result looks identical to a primary result in the output raster, but the tonnage now derives from a different observation epoch or a different sensor entirely. Unless the fallback event is recorded with the original failure context, the failure hash, and an explicit substitution justification, the double-counting and misattribution risk this is meant to prevent is instead concealed inside the manifest.

Deterministic Implementation Architecture

The implementation below captures provenance at every task boundary. It uses prefect for orchestration, rioxarray/xarray with dask for chunked raster I/O, rasterio and pyproj for explicit spatial operations, and structlog for audit-ready JSON telemetry. The ProvenanceTracker enforces an append-only lineage model: each node records the operation, input/output paths, parameter snapshot, active CRS, SHA-256 checksum, and status, and every task either emits a hashed artifact with a lineage node or routes through a logged fallback — there is no silent pass-through.

import hashlib
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Optional

import numpy as np
import rasterio
import rioxarray  # registers the xarray ".rio" accessor + "rasterio" engine
import xarray as xr
import pyproj
import structlog
from prefect import flow, task

# Structured, audit-ready JSON telemetry — one event per transformation boundary.
structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.JSONRenderer(),
    ]
)
log = structlog.get_logger()

# Validation gates — breaches raise rather than coercing a bad artifact downstream.
MAX_MASKED_FRACTION = 0.40       # reject scenes that lose >40% of pixels to masking
CANONICAL_CRS = "EPSG:4326"      # equal-area target resolved per project at ingestion


class ProvenanceTracker:
    """Append-only lineage recorder for MRV synchronization nodes."""

    def __init__(self, project_id: str, registry: str, canonical_crs: str):
        self.project_id = project_id
        self.registry = registry
        self.canonical_crs = canonical_crs
        self.lineage_nodes: List[Dict] = []

    def record_node(self, operation: str, inputs: List[str], outputs: List[str],
                    params: Dict, crs: str, checksum: Optional[str] = None,
                    status: str = "success") -> None:
        node = {
            "operation": operation,
            "inputs": inputs,
            "outputs": outputs,
            "parameters": params,
            "spatial_ref": crs,
            "output_checksum": checksum,
            "status": status,
            "recorded_at": datetime.now(timezone.utc).isoformat(),
        }
        self.lineage_nodes.append(node)
        log.info("lineage_node_recorded", operation=operation,
                 status=status, crs=crs, checksum=checksum)

    @staticmethod
    def compute_sha256(file_path: str) -> str:
        sha256 = hashlib.sha256()
        with open(file_path, "rb") as f:
            for chunk in iter(lambda: f.read(8192), b""):
                sha256.update(chunk)
        return sha256.hexdigest()

    def export_manifest(self, output_dir: Path) -> Path:
        manifest_path = output_dir / "provenance_manifest.json"
        manifest = {
            "project_id": self.project_id,
            "registry": self.registry,
            "canonical_crs": self.canonical_crs,
            "lineage_nodes": self.lineage_nodes,
        }
        manifest_path.write_text(json.dumps(manifest, indent=2))
        log.info("manifest_exported", path=str(manifest_path),
                 node_count=len(self.lineage_nodes))
        return manifest_path


@task
def align_and_mask(src_path: str, target_crs: str,
                   cloud_threshold: float = 0.15) -> Dict:
    """Cloud-mask, reproject once from the authoritative source, capture params."""
    with rasterio.open(src_path) as src:
        # Explicit datum declaration — reject untagged geometry at the door.
        if src.crs is None:
            raise ValueError(f"{src_path} has no CRS tag; refusing to guess a datum.")
        src_crs = pyproj.CRS.from_user_input(src.crs.to_string())

    # Lazy, chunked load so large tiles never blow the heap.
    ds = xr.open_dataset(src_path, engine="rasterio",
                         chunks={"x": 2048, "y": 2048})

    # Cloud masking via SWIR/Red band ratio (simplified Sentinel-2 example).
    cloud_mask = (ds["B11"] / ds["B04"]) < cloud_threshold
    masked_fraction = float(cloud_mask.mean().compute())
    if masked_fraction > MAX_MASKED_FRACTION:
        raise RuntimeError(
            f"masked fraction {masked_fraction:.2%} exceeds gate "
            f"{MAX_MASKED_FRACTION:.0%}; scene unusable")
    ds["carbon_proxy"] = ds["B11"].where(~cloud_mask, np.nan)

    # Single-pass reprojection FROM the source CRS — never from a derivative.
    ds = ds.rio.write_crs(src_crs)
    ds_aligned = ds.rio.reproject(target_crs)

    out_path = src_path.replace(".tif", "_aligned_masked.tif")
    ds_aligned.rio.to_raster(out_path, driver="GTiff", compress="DEFLATE")
    log.info("aligned_and_masked", output=out_path,
             masked_fraction=round(masked_fraction, 4),
             source_crs=str(src_crs), target_crs=target_crs)

    return {
        "output_path": out_path,
        "crs": target_crs,
        "params": {
            "cloud_threshold": cloud_threshold,
            "band_ratio": "B11/B04",
            "masked_fraction": round(masked_fraction, 4),
            "source_crs": str(src_crs),
        },
    }


@task
def compute_carbon_stock(raster_path: str,
                         fallback_path: Optional[str] = None) -> Dict:
    """Compute the carbon proxy; on failure, route to a logged fallback."""
    try:
        ds = xr.open_dataset(raster_path, engine="rasterio",
                             chunks={"x": 2048, "y": 2048})
        # Allometric scaling proxy: tC/ha = (proxy * 0.042) + 1.2 (example calibration).
        ds["tC_ha"] = ds["carbon_proxy"] * 0.042 + 1.2
        out_path = raster_path.replace("_aligned_masked.tif", "_carbon_stock.tif")
        ds["tC_ha"].rio.to_raster(out_path, driver="GTiff", compress="DEFLATE")
        log.info("carbon_stock_computed", output=out_path)
        return {"output_path": out_path, "status": "success",
                "failure_context": None}
    except Exception as exc:  # noqa: BLE001 — failure context is itself an artifact
        if fallback_path is None:
            log.error("carbon_stock_failed_no_fallback", error=str(exc))
            raise RuntimeError(f"computation failed, no fallback: {exc}") from exc
        # Substitution is never silent — preserve the original failure context.
        failure_hash = hashlib.sha256(str(exc).encode()).hexdigest()[:16]
        log.warning("fallback_routed", error=str(exc),
                    failure_hash=failure_hash, fallback=fallback_path)
        return {"output_path": fallback_path, "status": "fallback_routed",
                "failure_context": {"error": str(exc), "failure_hash": failure_hash}}


@flow(name="mrv_lineage_sync_flow")
def run_mrv_sync(project_id: str, registry: str, input_raster: str,
                 fallback_raster: str, work_dir: str) -> Path:
    tracker = ProvenanceTracker(project_id=project_id, registry=registry,
                                canonical_crs=CANONICAL_CRS)

    # Stage 1 — alignment & masking, with full parameter snapshot.
    align_result = align_and_mask(input_raster, CANONICAL_CRS)
    tracker.record_node(
        operation="cloud_mask_and_crs_align",
        inputs=[input_raster],
        outputs=[align_result["output_path"]],
        params=align_result["params"],
        crs=align_result["crs"],
        status="success",
    )

    # Stage 2 — carbon proxy with transparent fallback routing.
    stock_result = compute_carbon_stock(align_result["output_path"],
                                        fallback_path=fallback_raster)
    checksum = tracker.compute_sha256(stock_result["output_path"])
    tracker.record_node(
        operation="carbon_stock_computation",
        inputs=[align_result["output_path"]],
        outputs=[stock_result["output_path"]],
        params={"scaling_factor": 0.042, "intercept": 1.2,
                "failure_context": stock_result["failure_context"]},
        crs=CANONICAL_CRS,
        checksum=checksum,
        status=stock_result["status"],
    )

    return tracker.export_manifest(Path(work_dir))

Validation, Debugging & Compliance Mapping

Each design decision in the implementation maps to a specific regulatory control, which is what makes the manifest a submission artifact rather than a developer convenience.

  • Append-only nodes with SHA-256 checksums → ISO 14064-3 traceability. Because every node fixes the content hash of its output before the next stage reads it, a verifier can confirm that the raster they are auditing is the byte-identical artifact the manifest describes, satisfying the data-traceability and reproducibility expectations of ISO 14064-3 third-party validation.
  • Per-node parameter snapshots → Verra VM-series defensibility. Logging the cloud threshold, band ratio, masked fraction, and allometric calibration in each node gives auditors the evidence that thresholds were calibrated rather than guessed — the documentation standard Verra VM0042 and related methodologies demand for monitored parameters.
  • Explicit source-CRS capture and single-pass reprojection → geometric-stability requirements. Recording the source datum and reprojecting once from the authoritative tile keeps project-boundary geometry stable across monitoring periods, aligning with the geometric-integrity expectations carried over from CRS alignment and required for credit-volume consistency across cycles.
  • Transparent fallback with preserved failure context → CSRD ESRS E1 misstatement controls. The failure_hash and substitution justification let an auditor distinguish primary from substituted data, a direct control against the misstatement risk that CSRD ESRS E1 disclosures are scrutinized for, and a precondition for clean carbon credit registry data integration.

For debugging, treat the masked-pixel fraction and the distortion residual as monitored signals, not just pass/fail gates. Log them on every run, including the ones that pass, so a slowly drifting upstream export or a quietly updated grid file surfaces as a trend long before any single run breaches tolerance. Three recurring silent failures deserve dedicated diagnostics: a missing transformation grid that lets pyproj fall back to a null shift that looks successful, anti-meridian wrapping that inverts polygon area, and a fallback that fires so often it has become the de-facto primary path. Store the manifest under object-storage versioning with an immutability lock (for example AWS S3 Object Lock) alongside the registry submission so post-submission tampering is impossible. To standardize these events across heterogeneous platforms, emit them through a shared schema as described in tracking data lineage with OpenLineage for ESG audits.

Conclusion

MRV data lineage and provenance tracking is what converts raw geospatial processing into a regulator-ready evidence package. By recording an append-only node — operation, parameters, CRS, and content hash — at every transformation boundary, refusing untagged geometry, reprojecting once from the authoritative source, and routing every fallback through a logged, justified substitution, engineering teams eliminate the silent failures that historically trigger registry rejection. The result is a synchronization stage whose every tonne can be walked back to its source pixel and reproduced byte-for-byte by a third party. For the enterprise event-schema integration that standardizes these records across platforms, continue with Tracking Data Lineage with OpenLineage for ESG Audits.