Pipeline Orchestration & Compliance Reference #

Pipeline Orchestration & Compliance Reference is the control and governance plane of a Measurement, Reporting, and Verification (MRV) platform: the layer that decides when every stage runs, how it recovers from failure, which schema and reference data it is allowed to touch, and whether its outputs are fit to submit to a carbon registry. Where the other three sections of this site produce carbon numbers, this section is responsible for making those numbers reproducible, on-time, and defensible. It orchestrates the ingestion and normalization work described in MRV Architecture & Carbon Accounting Fundamentals, schedules the heavy raster processing built in Satellite Imagery Processing for Emissions Tracking, and gates the modeling outputs from Spatial Modeling & Carbon Stock Validation before a single tonne is claimed.

Treating orchestration and reference data as an afterthought is the most common reason otherwise-sound carbon pipelines fail third-party verification. A model can be scientifically correct and still be rejected because two runs produced different numbers from the same inputs, because an emission factor was silently upgraded mid-reporting-period, or because a partial backfill double-counted a region. This section addresses that risk head-on with three tightly coupled concerns: workflow orchestration that guarantees idempotent, retry-safe, observable execution; a canonical data-schema and reference layer — the shared Parquet contract and versioned emission-factor databases — that every stage reads from and writes to; and a compliance mapping layer that ties each output to the registry standards and methodologies it must satisfy (Verra VM-series, Gold Standard, CSRD ESRS E1, ISO 14064). The detailed engineering lives in three areas this page coordinates: orchestrating MRV data pipelines, the MRV data schema reference, and carbon registry standards and methodologies.

Core Pipeline Architecture #

The orchestration layer is not a scheduler that happens to run some scripts; it is the component that turns a collection of correct transformations into a reproducible system. Every MRV pipeline that survives audit shares the same structural spine: a directed acyclic graph (DAG) of stateless, idempotent tasks whose execution order, retry policy, and data contracts are declared explicitly and enforced by the orchestrator rather than assumed. The canonical stage sequence — ingest, normalize, factor, aggregate, verify — is deliberately linear and deterministic, because a reporting period must be reconstructable bit-for-bit from raw inputs and the pinned versions of the reference data used. The choice of engine (Prefect, Airflow, or Dagster) matters less than the guarantees the engine is configured to provide, and the trade-offs between them are worked through in detail under orchestrating MRV data pipelines.

Three properties are non-negotiable for a carbon pipeline. Idempotency means that re-running a task with the same inputs produces the same output and no duplicate side effects; without it, a retried backfill can double-count activity data or emit two credit-eligible records for one hectare. Retry safety means transient failures — a throttled STAC endpoint, a spot instance reclaimed mid-tile, a momentary object-store timeout — are absorbed by bounded, exponential retries at task granularity rather than failing an entire continental run. Observability means every task emits structured telemetry that becomes part of the audit trail: which inputs it read, which reference versions it pinned, how long it ran, and whether every validation gate passed. These are not operational niceties. They are the mechanism by which a registry, or a buyer’s due-diligence team, can trust that the number in a submission is the number the code actually produced.

The idempotency guarantee is implemented with an explicit idempotency key derived from the semantic identity of a unit of work — typically a hash over the project identifier, the reporting period, the tile or stratum, and the pinned versions of every reference input. The orchestrator uses that key to deduplicate: if a task with a given key has already completed and written its output, a re-run is a no-op that returns the existing artifact rather than recomputing and re-emitting it. This single discipline is what makes backfills, gap-fills, and disaster-recovery replays safe, and the patterns for constructing keys that survive partial and overlapping backfills are the subject of building idempotent backfills for carbon pipelines.

The Prefect flow below is a representative skeleton of the control plane. It orchestrates the full ingest-to-verify sequence, attaches retries and idempotency keys to every task, threads a run-scoped structlog logger through the graph, and places explicit validation gates that raise on failure rather than letting a defective artifact reach the aggregation and submission stages.

from __future__ import annotations

import hashlib
import json
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any

import geopandas as gpd
import pandas as pd
import pyarrow.parquet as pq
import structlog
from prefect import flow, task
from prefect.tasks import task_input_hash

structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.JSONRenderer(),
    ]
)
log = structlog.get_logger()

CANONICAL_CRS = "EPSG:6933"          # World Cylindrical Equal Area — area-honest pools
SCHEMA_CONTRACT_VERSION = "mrv-core-v3.2.0"
EF_DB_VERSION = "ipcc-2019-refinement+regional-2026.1"
REQUIRED_COLUMNS = {"geometry", "activity_id", "activity_qty", "unit", "epoch"}


class ValidationGateError(RuntimeError):
    """Raised when an artifact fails a hard compliance or integrity gate."""


def idempotency_key(*, project: str, period: str, stage: str) -> str:
    """Semantic identity of a unit of work; pins reference versions so a
    factor-database bump changes the key and forces a clean recompute."""
    material = "|".join([project, period, stage, SCHEMA_CONTRACT_VERSION, EF_DB_VERSION])
    return hashlib.sha256(material.encode("utf-8")).hexdigest()[:16]


@task(retries=3, retry_delay_seconds=[10, 30, 90], cache_key_fn=task_input_hash)
def ingest(source_uri: str, *, project: str, period: str) -> gpd.GeoDataFrame:
    """Pull raw activity/sensor features and tag them with provenance."""
    key = idempotency_key(project=project, period=period, stage="ingest")
    gdf = gpd.read_parquet(source_uri)
    gdf["_ingest_key"] = key
    gdf["_source_uri"] = source_uri
    log.info("ingest.complete", project=project, period=period,
             key=key, rows=len(gdf), source=source_uri)
    return gdf


@task(retries=3, retry_delay_seconds=[10, 30, 90])
def normalize(gdf: gpd.GeoDataFrame, *, project: str, period: str) -> gpd.GeoDataFrame:
    """Enforce the canonical CRS and the shared Parquet schema contract."""
    if gdf.crs is None:
        raise ValidationGateError("ingest emitted an untagged CRS; refusing to guess a datum")
    gdf = gdf.to_crs(CANONICAL_CRS)

    missing = REQUIRED_COLUMNS - set(gdf.columns)
    if missing:
        raise ValidationGateError(f"schema {SCHEMA_CONTRACT_VERSION} violated; missing {sorted(missing)}")

    invalid = int((~gdf.geometry.is_valid).sum())
    if invalid:
        log.warning("normalize.geometry.repaired", project=project, invalid=invalid)
        gdf["geometry"] = gdf.geometry.make_valid()

    log.info("normalize.complete", project=project, period=period,
             crs=CANONICAL_CRS, schema=SCHEMA_CONTRACT_VERSION, rows=len(gdf))
    return gdf


@task(retries=2, retry_delay_seconds=30)
def apply_factors(gdf: gpd.GeoDataFrame, ef_table: pd.DataFrame,
                  *, project: str, period: str) -> gpd.GeoDataFrame:
    """Join spatially explicit, versioned emission factors onto activity rows."""
    merged = gdf.merge(ef_table, on=["activity_id", "unit"], how="left", validate="many_to_one")
    unmatched = int(merged["ef_value"].isna().sum())
    if unmatched:
        raise ValidationGateError(
            f"{unmatched} rows had no factor in {EF_DB_VERSION}; "
            "unmapped activities cannot be quantified")
    merged["emissions_tco2e"] = merged["activity_qty"] * merged["ef_value"]
    merged["_ef_db_version"] = EF_DB_VERSION
    log.info("factor.complete", project=project, period=period,
             ef_db=EF_DB_VERSION, matched=len(merged))
    return merged


@task(retries=2, retry_delay_seconds=30)
def aggregate(gdf: gpd.GeoDataFrame, *, project: str, period: str) -> pd.DataFrame:
    """Roll pixel/parcel emissions up to the reporting stratum."""
    rolled = (gdf.groupby(["activity_id", "epoch"], as_index=False)["emissions_tco2e"]
              .sum().sort_values("emissions_tco2e", ascending=False))
    log.info("aggregate.complete", project=project, period=period,
             strata=len(rolled), total_tco2e=round(float(rolled["emissions_tco2e"].sum()), 3))
    return rolled


@task
def verify(rolled: pd.DataFrame, *, project: str, period: str,
           max_relative_uncertainty: float = 0.15) -> dict[str, Any]:
    """Hard verification gate: raises unless the batch is submission-ready."""
    if (rolled["emissions_tco2e"] < 0).any():
        raise ValidationGateError("negative emissions in aggregate; a factor or sign error upstream")
    if rolled.empty:
        raise ValidationGateError("empty aggregate; nothing to submit")

    total = float(rolled["emissions_tco2e"].sum())
    certificate = {
        "project": project,
        "period": period,
        "schema_contract": SCHEMA_CONTRACT_VERSION,
        "ef_db_version": EF_DB_VERSION,
        "crs": CANONICAL_CRS,
        "total_tco2e": round(total, 3),
        "max_relative_uncertainty": max_relative_uncertainty,
        "sealed_at": datetime.now(timezone.utc).isoformat(),
    }
    canonical = json.dumps(certificate, sort_keys=True, separators=(",", ":")).encode("utf-8")
    certificate["content_hash"] = hashlib.sha256(canonical).hexdigest()
    log.info("verify.passed", project=project, period=period,
             content_hash=certificate["content_hash"], total_tco2e=certificate["total_tco2e"])
    return certificate


@flow(name="mrv_reporting_flow", log_prints=True)
def run_reporting_period(source_uri: str, ef_uri: str, out_dir: str,
                         *, project: str, period: str) -> Path:
    """Orchestrate ingest -> normalize -> factor -> aggregate -> verify."""
    ef_table = pd.read_parquet(ef_uri)
    raw = ingest(source_uri, project=project, period=period)
    norm = normalize(raw, project=project, period=period)
    factored = apply_factors(norm, ef_table, project=project, period=period)
    rolled = aggregate(factored, project=project, period=period)
    certificate = verify(rolled, project=project, period=period)

    out = Path(out_dir) / f"{project}_{period}_submission.parquet"
    rolled.to_parquet(out, index=False)
    (out.with_suffix(".certificate.json")).write_text(json.dumps(certificate, indent=2))
    log.info("flow.complete", project=project, period=period, output=str(out))
    return out


if __name__ == "__main__":
    run_reporting_period(
        "s3://mrv/raw/activity.parquet",
        "s3://mrv/reference/ef_2026.1.parquet",
        "output/",
        project="REDD-BR-014",
        period=str(date(2026, 1, 1)),
    )

Every task in this skeleton follows the same contract that runs through the whole platform: bounded retries on transient failure, an idempotency key that pins the reference versions it consumed, structured telemetry on entry and exit, and a hard gate that raises rather than emitting a defective artifact. The verify task does not “clean up” bad data — it refuses it, so a defect surfaces as a failed run an operator investigates instead of a silently corrupted submission an auditor later rejects.

Spatial Alignment & Data Normalization #

The normalize stage is where the reference layer stops being documentation and becomes an enforced contract. In an MRV platform, dozens of upstream producers — optical compositing jobs, LiDAR/SAR fusion runs, activity-data connectors, field-inventory ingest — all write into the same downstream aggregation. If each producer chooses its own column names, CRS conventions, units, and null handling, the aggregation stage becomes a swamp of defensive special-casing and the audit trail becomes impossible to reason about. The canonical Parquet schema exists to eliminate that entropy: it defines the exact column set, dtypes, unit conventions, CRS policy, and provenance fields that every artifact crossing a stage boundary must satisfy, and it is versioned so a schema change is a deliberate, dated event rather than a silent drift. The full field-by-field data dictionary is maintained in the MRV data schema reference, and the specific column-level definitions live in the canonical Parquet schema data dictionary for MRV.

CRS handling is the highest-value gate in the normalize stage because CRS errors are silent and systematic. A dataset delivered in unprojected WGS84 and naively treated as metres, or a subtle datum shift between two nominally identical EPSG:4326 layers, will not throw — it will quietly bias every area-based calculation downstream. The normalize stage therefore refuses untagged geometry outright, reprojects everything to a single area-honest grid (EPSG:6933 for global pools, or a jurisdiction-appropriate equal-area projection), and records the transformation it applied. The mechanics of doing this correctly, including the failure modes that produce silent datum shifts, are covered in the foundational geospatial coordinate reference systems (CRS) alignment work that this stage consumes and enforces.

Reference data — emission factors, allometric equations, biome-specific expansion factors — is the second half of the normalization problem, and it is where reproducibility is most often lost. An emission-factor database is not static: IPCC refinements, regional recalibrations, and methodology updates all change factor values over time. If a pipeline reads “the current factors” at run time, then two runs of the same reporting period separated by a database update will produce different numbers from identical activity data, and neither run can be reconstructed later. The discipline that prevents this is strict version pinning: every run records the exact emission-factor database version it used, that version is immutable, and the idempotency key incorporates it so a factor upgrade forces a clean, traceable recompute rather than a silent overwrite. The practices for versioning, storing, and retiring factor databases so that historical reporting periods remain reproducible are detailed in versioning emission factor databases for reproducible MRV. Because factors are spatially explicit rather than globally averaged, the normalization stage also carries the per-region variance that feeds downstream uncertainty propagation, connecting directly to the modeling work in Spatial Modeling & Carbon Stock Validation.

Compliance Mapping & Regulatory Framework #

The compliance layer answers a question the modeling layers cannot: is this output acceptable to the standard we are reporting under? Scientific correctness is necessary but not sufficient — a carbon number must also satisfy the specific stratification rules, uncertainty deductions, permanence tests, and disclosure formats of the registry or regulation it targets. The governance plane treats those requirements as machine-checkable controls embedded in the graph, not as prose narrated in a methodology PDF and hoped for at audit time. The full mapping between pipeline outputs and each standard is maintained under carbon registry standards and methodologies.

• Verra VM-series. Project-crediting methodologies such as VM0047 (afforestation, reforestation and revegetation) and VM0042 (improved agricultural land management) impose explicit stratification, plot-density, and uncertainty-deduction rules. The verification stage encodes these as deductions applied before tonnage is issued, so an over-optimistic estimate is conservatively discounted rather than submitted at face value. The GIS-specific requirements — boundary topology, stratum delineation, sampling design — differ between registries, and the contrasts are worked through in Verra VM0047 vs Gold Standard GIS requirements.
• Gold Standard. Gold Standard methodologies share the conservativeness principle but differ in their spatial-data expectations, monitoring cadence, and safeguarding requirements. The compliance layer parameterizes these differences rather than forking the pipeline, so a single platform can serve multiple registries from one canonical data model.
• CSRD ESRS E1. The Corporate Sustainability Reporting Directive’s climate standard demands granular, assured disclosures — including gross Scope 1/2/3 emissions, removals, and the estimation uncertainty behind them. Meeting it means carrying uncertainty bands and lineage on every published figure, and formatting disclosures to the ESRS taxonomy. The specific crosswalk from spatial MRV outputs to ESRS E1 datapoints is set out in mapping CSRD ESRS E1 disclosures to spatial MRV outputs.
• ISO 14064. ISO 14064-2 governs project-level quantification (baseline, additionality, leakage, permanence) and ISO 14064-3 governs validation and verification, emphasizing reproducibility and conservativeness. The pipeline satisfies both by making every figure recomputable from pinned inputs and by biasing every uncertainty treatment toward the conservative side of the distribution.

The compliance layer also reconciles project-level land-sector removals against corporate inventories to prevent double counting, drawing on the boundary definitions established in GHG Protocol Scope 3 spatial mapping. A removal claimed by a landowner and an emission reduction claimed in a buyer’s value chain must not silently refer to the same hectare, and the spatial-join logic that prevents this overlap is a compliance control as much as an engineering one.

Audit Trails, Lineage & Provenance #

Verification is a question about evidence, and the orchestration layer is where that evidence is generated as a by-product of running the pipeline rather than assembled by hand afterward. Every task in the flow emits structured telemetry, and the verify stage seals a tamper-evident certificate whose content hash chains over the reporting total, the schema-contract version, the emission-factor database version, the CRS, and the timestamp. Any later mutation of those inputs changes the digest, so a certificate is a cryptographic assertion that a specific number was produced from a specific configuration. This inherits directly from the provenance discipline established in MRV data lineage and provenance tracking, which this section operationalizes at the orchestration level.

A complete audit record for one reporting period captures four things, and the orchestrator is responsible for ensuring none of them can be omitted:

1. Input identity — content checksums, STAC item IDs, and temporal stamps for every raster, vector, and table the run consumed, plus the source URIs recorded at ingest.
2. Pinned configuration — the schema-contract version, the emission-factor database version, the CRS, the library and code versions, and any random seed used for a Monte Carlo pass.
3. Validation outcomes — the pass/fail result of every gate, the topology checks, the uncertainty bounds, and every compliance deduction applied.
4. Output artifacts — the GeoParquet and Cloud-Optimized GeoTIFF URLs, their hashes, and the sealed verification certificate that authorizes submission.

Because outputs land in append-only object storage with immutable retention, a recalibration never overwrites history — it produces a new, versioned correction layer alongside the original, and both carry their own certificates. This is what makes lineage queryable rather than archival: an auditor can walk from a retired credit back to the exact run, the exact factor version, and the exact tiles that produced it. When broken or missing lineage does surface — a dropped parent_run_id, an unversioned reference read, a gap in the hash chain — the diagnostic playbook in troubleshooting broken data lineage in MRV audits is the reference for reconstructing it. The sealed certificate is also what closes the loop into the registry: it is submitted alongside tonnage so the registry, and any downstream buyer, can independently recompute the figure.

Production Deployment & Validation Patterns #

Running this control plane in production is an infrastructure discipline in its own right. The orchestrator is the single point at which idempotency, retries, per-task CRS gates, and telemetry are enforced, so the deployment patterns around it are what keep the guarantees intact under real load, partial failure, and change over time.

• Orchestrator selection by guarantee, not fashion. Prefect, Airflow, and Dagster all express DAGs, but they differ in dynamic task mapping, native retries, caching, and typed data passing. Choose the engine whose defaults make idempotency and observability the path of least resistance; the head-to-head comparison for MRV workloads is in Prefect vs Airflow vs Dagster for MRV pipelines.
• Idempotent backfills as a first-class operation. Reporting corrections, late-arriving activity data, and reprocessing after a factor upgrade all require backfills that do not double-count. Idempotency keys that incorporate the reporting period and pinned reference versions make overlapping backfills safe, as detailed in building idempotent backfills for carbon pipelines.
• Schema and factor versioning under CI. Treat the canonical schema and the emission-factor database as versioned, tested artifacts. A schema migration or factor bump runs through the same review, changelog, and regression-test gate as code, so no reporting figure changes without a dated, reviewed cause.
• Data-quality gates as executable contracts. Layer declarative expectations — with great_expectations or an equivalent — over the schema contract so range, null, and distribution violations fail the run at the boundary rather than surfacing as anomalies in a submission. The pattern is developed in building Great Expectations checks for emissions data.
• Observability and cost control. Instrument the orchestrator with metrics — task latency, retry rates, validation-failure rates, and cloud egress — and route the structured logs to a queryable store. Use spot capacity for stateless raster algebra and reserved capacity for the persistent reference and topology databases, and alert on drift in the median uncertainty and gate-failure rates even on runs that pass.

For teams standardizing on open specifications, the OpenLineage standard provides a portable lineage model that plugs directly into these orchestrators, while the STAC Specification keeps input identity machine-readable across the ingest boundary. Both decouple the governance plane from any single engine, which is what lets a platform migrate orchestrators without losing its audit history.

Conclusion #

Pipeline Orchestration & Compliance Reference is the layer that makes carbon numbers trustworthy: it schedules the work, guarantees that re-running it is safe and reproducible, enforces a single canonical schema and pinned reference data, and gates every output against the registry standard it must satisfy before it is submitted. By treating orchestration, the reference layer, and compliance mapping as one coordinated control plane — idempotent tasks, versioned factors, hash-sealed certificates, and machine-checkable standards mapping — engineering teams turn scientifically correct models into auditable, submission-ready carbon accounting. The detailed engineering continues in the three areas this page coordinates: orchestrating MRV data pipelines for the scheduler, retry, and backfill patterns; the MRV data schema reference for the canonical Parquet contract and factor versioning; and carbon registry standards and methodologies for the Verra, Gold Standard, CSRD, and ISO controls every output must clear. Together with the modeling, imagery, and architecture sections it governs, this control plane is what makes a scalable, transparent, and defensible carbon market operationally real.

Related guides #

• MRV Architecture & Carbon Accounting Fundamentals — the foundational ingestion, CRS, and lineage layer this plane schedules and enforces.
• Satellite Imagery Processing for Emissions Tracking — the raster-processing workloads the orchestrator runs at scale.
• Spatial Modeling & Carbon Stock Validation — the modeling outputs this plane gates before submission.
• Orchestrating MRV Data Pipelines — scheduler, retry, idempotency, and backfill patterns for the control plane.
• Carbon Registry Standards & Methodologies — the Verra, Gold Standard, CSRD ESRS E1, and ISO 14064 controls each output must satisfy.