Prefect vs Airflow vs Dagster for MRV Pipelines #

Choosing an orchestrator for measurement, reporting, and verification (MRV) work is not a question of which tool schedules DAGs most elegantly; it is a question of which one lets you regenerate a two-year emissions figure, prove where every number came from, and survive a verifier who asks to replay the pipeline from raw tiles. This decision guide sits within Orchestrating MRV Data Pipelines, the coordination discipline inside the wider Pipeline Orchestration & Compliance Reference stack, and it assumes you have already read why building idempotent backfills for carbon pipelines is the property every candidate must support before it is even eligible.

The three mainstream Python orchestrators — Apache Airflow, Prefect, and Dagster — are not interchangeable for carbon work, and the honest answer is not “it depends”. Each optimises for a different shape of problem: Airflow for scheduled, operationally hardened batch; Prefect for dynamic, Pythonic fan-out with minimal ceremony; and Dagster for data-asset lineage as a first-class runtime concern. MRV pipelines stress exactly the three axes on which they diverge most — dynamic fan-out over thousands of satellite tiles, machine-readable lineage for audit, and deterministic replay of historical windows — so the fit is decidable if you score the tools against those axes rather than against a generic feature list.

Why orchestrator choice matters for MRV #

MRV pipelines fail verification for orchestration reasons far more often than for modelling reasons. Three structural pressures make the choice consequential rather than cosmetic.

First, the unit of work is the tile-month, and there are a lot of them. A national forest-carbon programme partitions its area of interest into thousands of Sentinel-2 tiles and processes each across many monthly windows, so a single reporting run fans out into tens of thousands of independent tasks whose cardinality changes every period as tiles enter and leave the acquisition schedule. An orchestrator that expands its task graph statically at parse time, or that struggles to map over a runtime-determined list, forces you to either over-provision the graph or fracture the pipeline into brittle sub-DAGs. This is where Prefect’s runtime task mapping and Dagster’s dynamic partitions pull ahead of a naive Airflow implementation.

Second, replay is a compliance requirement, not an operational nicety. When a verifier disputes a figure, or an upstream emission-factor table is revised, you must recompute a specific historical window and get an answer that is identical to — or transparently reconcilable with — the original. That demands idempotent, deterministically-keyed tasks and clean backfill semantics, the property examined in depth under building idempotent backfills for carbon pipelines. An orchestrator whose retry or backfill mechanics can silently double-write a partition is disqualifying, regardless of how good its UI looks.

Third, lineage is evidence. Under ISO 14064-3 and the major registry methodologies, the auditor traces each reported number back through every transformation to the raw observation. An orchestrator that treats data assets and their lineage as first-class runtime objects — as Dagster does with software-defined assets — hands you that trace for free, whereas a task-centric orchestrator makes you reconstruct it after the fact from logs and an external lineage collector. The gap between “lineage emitted by the runtime” and “lineage reassembled from logs” is precisely the gap a verifier probes.

Pre-flight: selection criteria and a scoring table #

Before comparing, fix the criteria and mark which are non-negotiable. Two are gating: every candidate must support idempotent, deterministically-keyed tasks and must be able to emit machine-readable lineage (natively or via OpenLineage). A tool that fails either is eliminated before convenience features are weighed. The remaining criteria are trade-offs you score by team profile: dynamic fan-out over tiles, data-asset/lineage awareness, retry and idempotency ergonomics, backfill semantics, typing and data contracts, local reproducibility, observability, and ecosystem breadth.

The table scores each orchestrator against those criteria on a compact scale — strong, adequate, or weak — with a one-clause justification. The scores reflect MRV fit specifically, not general-purpose merit.

Read the table by column, not by counting ticks. Airflow’s strengths cluster in operational maturity and backfill controls; its weaknesses are exactly the MRV-critical ones of native lineage and data contracts. Prefect trades some of Airflow’s operational surface for markedly better dynamic fan-out and idempotency ergonomics. Dagster is the only column with “strong” against both lineage and data contracts, which is why it wins when audit evidence is the binding constraint — at the cost of a steeper conceptual model.

Deterministic transformation logic: the same MRV task in Prefect and Dagster #

The clearest way to feel the difference is to implement one idempotent MRV task in two of the three tools. The task: derive land-use-change activity data for a single satellite tile and month, reproject to an equal-area grid so per-tile areas are honest, and write to a deterministically-keyed path so a replay never double-writes. Both implementations use structlog for audit-ready telemetry and refuse untagged geometry.

The Prefect version leans on runtime mapping for fan-out and cache_key_fn for idempotency — identical inputs return the cached result rather than recomputing.

import hashlib
from datetime import timedelta

import geopandas as gpd
import structlog
from prefect import flow, task, unmapped
from prefect.tasks import task_input_hash

log = structlog.get_logger()

CANONICAL_CRS = "EPSG:6933"  # equal-area grid so per-tile areas are honest


def _partition_key(tile_id: str, period: str) -> str:
    # Deterministic key: same (tile, month) -> same output path, always.
    return hashlib.sha256(f"{tile_id}:{period}".encode()).hexdigest()[:16]


@task(
    retries=3,
    retry_delay_seconds=30,
    cache_key_fn=task_input_hash,          # identical inputs -> cache hit, no re-run
    cache_expiration=timedelta(days=7),
)
def compute_tile_activity(tile_id: str, period: str, src_crs: str) -> str:
    """Idempotent land-use-change activity for one tile-month.

    Re-running with the same (tile_id, period) writes to a deterministic path,
    so a backfill can replay any window without duplicating rows.
    """
    key = _partition_key(tile_id, period)
    out_path = f"s3://mrv-activity/{period}/{tile_id}/{key}.parquet"

    gdf = gpd.read_parquet(f"s3://mrv-tiles/{period}/{tile_id}.parquet")
    if gdf.crs is None:
        raise ValueError(f"{tile_id}: untagged geometry; refusing to guess a datum")
    # Enforce declared source CRS, then reproject to the equal-area canonical grid.
    gdf = gdf.set_crs(src_crs, allow_override=False).to_crs(CANONICAL_CRS)

    gdf["area_ha"] = gdf.geometry.area / 10_000.0
    activity = gdf.dissolve(by="lulc_class", aggfunc={"area_ha": "sum"}).reset_index()
    activity[["tile_id", "period", "partition_key"]] = tile_id, period, key

    activity.to_parquet(out_path, index=False)
    log.info("tile.activity.written", tile_id=tile_id, period=period,
             partition_key=key, crs=CANONICAL_CRS, classes=int(activity.shape[0]),
             out_path=out_path)
    return out_path


@flow(name="mrv_tile_activity")
def run(period: str, tile_ids: list[str], src_crs: str = "EPSG:4326") -> list[str]:
    # Dynamic fan-out: one mapped task per tile, cardinality decided at runtime.
    futures = compute_tile_activity.map(
        tile_ids, period=unmapped(period), src_crs=unmapped(src_crs))
    return [f.result() for f in futures]

The Dagster version models the same computation as a software-defined asset partitioned by tile. The (asset, partition) pair is the identity, so re-materialising is inherently idempotent, and the output metadata Dagster records becomes queryable lineage without a separate collector.

import hashlib

import geopandas as gpd
import structlog
from dagster import (
    asset, AssetExecutionContext, StaticPartitionsDefinition,
    RetryPolicy, MetadataValue,
)

log = structlog.get_logger()
CANONICAL_CRS = "EPSG:6933"  # equal-area grid so per-tile areas are honest

tiles = StaticPartitionsDefinition(["N00E010", "N00E020", "N10E010"])


@asset(
    partitions_def=tiles,          # one materialisation per tile -> native fan-out
    retry_policy=RetryPolicy(max_retries=3, delay=30),
    code_version="1.3.0",          # bump invalidates stale materialisations on replay
    group_name="mrv_activity",
)
def tile_activity(context: AssetExecutionContext) -> None:
    """Land-use-change activity for one tile partition.

    Dagster tracks the (asset, partition) identity, so re-materialising a
    partition is idempotent and lineage is recorded by the runtime itself.
    """
    tile_id = context.partition_key
    period = context.run.tags.get("period", "unknown")
    key = hashlib.sha256(f"{tile_id}:{period}".encode()).hexdigest()[:16]

    gdf = gpd.read_parquet(f"s3://mrv-tiles/{period}/{tile_id}.parquet")
    if gdf.crs is None:
        raise ValueError(f"{tile_id}: untagged geometry; refusing to guess a datum")
    gdf = gdf.to_crs(CANONICAL_CRS)  # source CRS trusted from the tagged tile
    gdf["area_ha"] = gdf.geometry.area / 10_000.0
    activity = gdf.dissolve(by="lulc_class", aggfunc={"area_ha": "sum"}).reset_index()

    out_path = f"s3://mrv-activity/{period}/{tile_id}/{key}.parquet"
    activity.to_parquet(out_path, index=False)

    log.info("tile.activity.materialised", tile_id=tile_id, period=period,
             partition_key=key, crs=CANONICAL_CRS, classes=int(activity.shape[0]))
    # Materialisation metadata becomes queryable lineage / audit evidence.
    context.add_output_metadata({
        "partition_key": MetadataValue.text(key),
        "crs": MetadataValue.text(CANONICAL_CRS),
        "lulc_classes": MetadataValue.int(int(activity.shape[0])),
        "out_path": MetadataValue.path(out_path),
    })

The two are functionally equivalent, but the ergonomic difference is instructive. In Prefect you keep the deterministic key and idempotency logic in your own hands — flexible, and honest about what you are guaranteeing. In Dagster the partition is the key, the runtime records the materialisation, and the metadata you attach is lineage the moment it is written. An Airflow implementation would sit between them: a PythonOperator running the same body, idempotency entirely your responsibility, and lineage emitted through an OpenLineage integration configured alongside the DAG. All three write the same equal-area Parquet; they differ in how much audit infrastructure you get without building it.

Compliance gating: lineage and audit evidence per orchestrator #

The audit question is always the same — can you reconstruct, for any reported figure, the exact inputs, code version, and transformations that produced it? The orchestrators answer it with different amounts of built-in support, and the difference decides how much bespoke lineage plumbing your team maintains.

• Airflow is task-centric and emits no data lineage natively. In practice you attach the OpenLineage integration, which captures input/output datasets and job facets per task run, and you feed those facets into your evidence store. This is a well-trodden path that connects cleanly to tracking data lineage with OpenLineage for ESG audits, but the lineage graph lives outside Airflow and its completeness depends on disciplined instrumentation of every operator.
• Prefect records structured run and task events with input hashes, and its caching keys already tie a result to its inputs, which gives you a strong reproducibility signal. Asset-level lineage is newer and less central than in Dagster, so for audit you typically serialise the run metadata and partition keys yourself — the structlog events in the code above are designed for exactly that hand-off.
• Dagster treats the data asset, its code version, its partitions, and its upstream/downstream edges as runtime objects. The materialisation event, the metadata you attach, and asset checks form a lineage graph the platform maintains for you. When a verifier asks which raw tiles fed a disputed tile-month, the answer is a query against the asset catalogue rather than a log-scraping exercise.

Whichever orchestrator you pick, the outputs must land in the shape the rest of the compliance stack expects. That contract is defined by the MRV data schema reference: the canonical Parquet columns, the partition layout, and the metadata keys an auditor traces. The orchestrator choice changes how lineage is produced; it does not change the schema the evidence must conform to.

Production integration: a decision procedure and recommendation by team #

Turn the analysis into a repeatable choice with a fixed procedure, then read off the recommendation for your team profile.

1. Fix the non-negotiables. Confirm each candidate can express idempotent, deterministically-keyed tasks and emit machine-readable lineage. Eliminate anything that fails a gate before weighing convenience.
2. Diagnose the lineage requirement. If auditors demand asset-level lineage and freshness as a runtime property, weight Dagster heavily. If lineage can be an attached integration, keep all three in play.
3. Diagnose the fan-out shape. If dynamic fan-out over thousands of tiles with variable cardinality dominates, favour Prefect’s runtime mapping or Dagster’s dynamic partitions over a static Airflow graph.
4. Weigh team maturity and operations. Score each team against operational hardening (Airflow), Python-first velocity (Prefect), and readiness to model software-defined assets and data contracts (Dagster).
5. Prototype the backfill. Implement one idempotent tile-month task in the shortlist and replay a historical window; confirm byte-identical output and clean partition semantics — the fan-out that makes this tractable is the same async satellite tile processing with Dask that sits under the compute layer.
6. Wire audit evidence and commit. Serialise run metadata, partition keys, and lineage to the submission store, then standardise on whichever tool produced the cleanest reproducible trail.

The recommendation matrix by team profile:

• Established data-platform team with strong ops and a large provider surface, lineage acceptable as an integration: choose Airflow with OpenLineage. Its backfill maturity and operator ecosystem are unmatched, and the lineage gap is closable with disciplined instrumentation.
• Small-to-mid Python-first team, tile fan-out heavy, wants velocity and clean idempotency without running a heavy scheduler: choose Prefect. Runtime mapping and cache_key_fn fit MRV fan-out and replay with minimal ceremony.
• Team where audit lineage is the binding constraint and members are ready to model data assets and contracts: choose Dagster. Software-defined assets, typed IO, and asset checks make the audit trace a runtime artifact rather than a reconstruction — the amber path in the diagram, chosen precisely because it ships audit-ready lineage.

There is no universally correct orchestrator, but for a given MRV team there is a defensible one. Score the tools against dynamic fan-out, lineage, and idempotent replay; prototype a single tile-month backfill in the shortlist; and let the cleanest reproducible audit trail — not the prettiest UI — decide.

Related guides #

• Orchestrating MRV Data Pipelines — the parent coordination discipline this decision sits within.
• Building Idempotent Backfills for Carbon Pipelines — the replay property every candidate orchestrator must support.
• MRV Data Schema Reference — the canonical output contract every orchestrator’s tasks must conform to.
• Async Satellite Tile Processing with Dask — the tile-partitioned compute layer the orchestrator fans out over.