Spatial Pipeline EngineeringMRV Automation

Great Expectations vs Pandera vs Soda for MRV Validation

Choosing a validation framework for carbon work is not a question of which library has the most expectations. It is a question of which one produces evidence an auditor accepts, expresses the checks that geospatial carbon data actually needs, and survives being run ten thousand times a night over tile partitions. This decision guide sits within emissions data quality and validation gates in the MRV architecture and carbon accounting fundamentals stack, and it assumes the gate taxonomy from that page: a small set of blocking invariants and a larger set of trended indicators.

The three mainstream Python options optimise for different things. Great Expectations is a validation platform with a documentation and evidence layer attached. Pandera is a schema library that puts validation in the type system next to the code. Soda is a monitoring product that watches datasets in a warehouse on a schedule. Those are three genuinely different shapes, and MRV work stresses them unevenly — because the checks that matter most in carbon pipelines are geometric, unit-aware, and cross-partition, and none of the three models those natively.

Where each framework sits relative to the pipeline and the evidence store Three placements shown against a pipeline. Pandera sits inside the transformation code as a decorator on the function boundary, so it validates in-process on every call and fails the task. Great Expectations sits at the stage boundary as a separate validation run over the written artefact, producing a versioned result document. Soda sits outside the pipeline entirely, scanning the warehouse table on a schedule after the fact. A panel notes that only the first two can block promotion, that only the second produces a durable evidence document by default, and that the third detects problems after the data has already been read by someone. Three different places in the pipeline, not three versions of the same thing Transform in-process function Write artefact to the curated zone Warehouse table read by consumers Pandera Great Expectations Soda What the placement decides Pandera and GE can block promotion; Soda cannot. GE produces a durable result document by default. Soda finds the problem after someone has read the data. The frameworks are complements more often than alternatives — but only one of them can be the blocking gate. Pick that one first, then decide whether the others earn their operational cost.

Root Cause Analysis

The reason this choice is harder for MRV than for ordinary analytics is that the checks that matter are not the checks these frameworks were designed around.

Geometry is not a column type any of them models. Every one of the three validates tabular values well and none has a native concept of a coordinate reference system, geometry validity, or an anti-meridian crossing. Those are the checks that catch the failures that actually corrupt carbon numbers, so whichever framework you pick, the geospatial invariants will be custom code invoked through its extension mechanism. How pleasant that extension mechanism is turns out to matter more than the built-in expectation catalogue.

Units are semantics, not schema. A column typed as float64 with a plausible range passes every default check while being in the wrong unit, and unit errors are among the most common and most damaging defects in emissions data. Validating units means asserting metadata alongside values — the Parquet field metadata described in the MRV data schema reference — which again is custom.

The most important check is about a set, not a row. Partition completeness — did this run write every tile-month it should have — cannot be expressed as a row-level or even a table-level expectation, because the expected set comes from outside the data. Frameworks that assume validation is a function of a dataframe make this awkward; the check ends up living beside them rather than inside them.

The consequence is that framework choice determines the ergonomics and the evidence, not the coverage. Coverage comes from what you write.

Scoring the three frameworks against MRV-specific criteria A comparison across six criteria. On failing the run at the point of computation, Pandera is strong, Great Expectations is adequate, and Soda is weak. On producing a durable evidence document, Great Expectations is strong, Pandera is weak, and Soda is adequate. On custom geospatial checks, Pandera is strong through plain Python checks, Great Expectations is adequate through custom expectations, and Soda is weak. On cross-partition and completeness checks, all three are weak and the check lives outside the framework. On per-partition overhead at ten thousand invocations, Pandera is strong, Great Expectations is adequate, and Soda is not applicable. On non-engineer authorship of checks, Soda is strong through its YAML syntax, Great Expectations is adequate, and Pandera is weak. Scored against what MRV actually needs Read by row, not by counting ticks. Pandera Great Expectations Soda Fails the run where the computation happens strong adequate weak Durable evidence document for a verifier weak strong adequate Custom geospatial checks (CRS, validity, area) strong adequate weak Cross-partition completeness weak weak weak Overhead at 10 000 invocations per night strong adequate n/a Checks authored by non-engineers weak adequate strong

Diagnostic Pipeline / Pre-Flight Validation

Before choosing, write the four checks that matter most in your pipeline and see how each framework expresses them. The exercise takes an afternoon and settles the question faster than any feature comparison. The four are a unit assertion from metadata, a geometry validity and CRS check, a distributional check against the previous run, and a partition-completeness check against an externally computed set.

from dataclasses import dataclass

import geopandas as gpd
import pyarrow.parquet as pq
import structlog

log = structlog.get_logger()

CANONICAL_CRS = "EPSG:6933"
EXPECTED_UNITS = {"activity_value": "MJ", "co2e_tonnes": "t", "area_ha": "ha"}


@dataclass(frozen=True)
class CheckResult:
    name: str
    passed: bool
    blocking: bool
    detail: dict


def check_units(path: str) -> CheckResult:
    """The check no framework does natively: units live in field metadata, and a
    float column in the wrong unit passes every value-range expectation."""
    schema = pq.read_schema(path)
    found, wrong = {}, {}
    for field in schema:
        meta = field.metadata or {}
        unit = meta.get(b"unit", b"").decode() or None
        if field.name in EXPECTED_UNITS:
            found[field.name] = unit
            if unit != EXPECTED_UNITS[field.name]:
                wrong[field.name] = {"declared": unit, "expected": EXPECTED_UNITS[field.name]}

    passed = not wrong and all(found.get(c) for c in EXPECTED_UNITS)
    if not passed:
        log.error("validation.units", wrong=wrong, found=found)
    return CheckResult("units_declared_and_correct", passed, True,
                       {"found": found, "wrong": wrong})


def check_geometry(gdf: gpd.GeoDataFrame) -> CheckResult:
    """Three geospatial invariants no tabular framework models."""
    detail = {}
    if gdf.crs is None:
        return CheckResult("geometry_valid", False, True, {"reason": "no_crs"})

    detail["crs"] = gdf.crs.to_string()
    detail["invalid"] = int((~gdf.geometry.is_valid).sum())
    detail["empty"] = int(gdf.geometry.is_empty.sum())

    geographic = gdf.to_crs("EPSG:4326").bounds
    detail["antimeridian"] = int(((geographic["maxx"] - geographic["minx"]) > 180).sum())

    passed = (detail["crs"] == CANONICAL_CRS and detail["invalid"] == 0
              and detail["empty"] == 0 and detail["antimeridian"] == 0)
    if not passed:
        log.error("validation.geometry", **detail)
    return CheckResult("geometry_valid", passed, True, detail)


def check_completeness(written: set[str], expected: set[str]) -> CheckResult:
    """A statement about a SET, computed outside the data. No framework models
    this, because none of them can know what should have been written."""
    missing = expected - written
    extra = written - expected
    passed = not missing and not extra
    if not passed:
        log.error("validation.completeness", missing=len(missing), extra=len(extra),
                  example_missing=sorted(missing)[:3])
    return CheckResult("partition_completeness", passed, True,
                       {"expected": len(expected), "written": len(written),
                        "missing": len(missing), "extra": len(extra)})


def check_distribution(current: dict, previous: dict | None,
                       tolerance: float = 0.25) -> CheckResult:
    """An indicator, not an invariant — it drifts legitimately, so it flags
    rather than blocks."""
    if previous is None:
        return CheckResult("distribution_stable", True, False, {"baseline": True})

    moves = {}
    for key in ("value_p50", "value_p99", "null_rate"):
        prev, cur = previous.get(key), current.get(key)
        if prev in (None, 0) or cur is None:
            continue
        rel = abs(cur - prev) / abs(prev)
        if rel > tolerance:
            moves[key] = {"previous": prev, "current": cur, "relative": round(rel, 3)}

    if moves:
        log.warning("validation.distribution_moved", moves=moves)
    return CheckResult("distribution_stable", not moves, False, {"moves": moves})

Note what that code implies. Two of the four checks — units and completeness — are framework-independent by nature, and a third is trivial plain Python over a GeoDataFrame. What a framework buys you is the harness: how results are collected, how failures are reported, and what artefact survives the run.

Deterministic Transformation Logic

The same suite expressed through each framework shows the ergonomic difference clearly. Pandera puts the schema in the type system; Great Expectations puts it in a suite that produces a result document; Soda puts it in YAML evaluated against a warehouse table.

# --- Pandera: validation as a type annotation on the function boundary --------
import pandera.pandas as pa
from pandera.typing import DataFrame, Series


class MrvActivity(pa.DataFrameModel):
    """The schema IS the contract, and it fails where the computation happens."""
    activity_id: Series[str] = pa.Field(nullable=False, unique=True)
    activity_value: Series[float] = pa.Field(ge=0, le=1e9, nullable=False)
    area_ha: Series[float] = pa.Field(gt=0, le=1e7, nullable=False)
    co2e_tonnes: Series[float] = pa.Field(ge=0, nullable=True)
    factor_version: Series[str] = pa.Field(nullable=False)

    @pa.dataframe_check
    def co2e_requires_factor(cls, df) -> bool:
        # Cross-column semantics: a tonnage without a pinned factor version is
        # unreproducible, and no per-column rule can express that.
        return not (df["co2e_tonnes"].notna() & df["factor_version"].isna()).any()

    class Config:
        strict = True          # reject unexpected columns — schema drift fails loudly
        coerce = False         # never silently cast; a wrong dtype is a defect


@pa.check_types(lazy=True)
def apply_factors(df: DataFrame[MrvActivity]) -> DataFrame[MrvActivity]:
    """Validated on entry and on exit, in-process, at the point of computation."""
    return df
# --- Great Expectations: a suite that yields a durable result document --------
import great_expectations as gx

def build_suite(context) -> None:
    suite = context.suites.add(gx.ExpectationSuite(name="mrv_activity"))
    suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="activity_id"))
    suite.add_expectation(gx.expectations.ExpectColumnValuesToBeUnique(column="activity_id"))
    suite.add_expectation(
        gx.expectations.ExpectColumnValuesToBeBetween(
            column="area_ha", min_value=0, max_value=1e7, strict_min=True))
    suite.add_expectation(
        gx.expectations.ExpectColumnValuesToBeInSet(
            column="activity_unit", value_set=["MJ", "kWh", "t", "ha"]))
    # The evidence value is here: the validation result is a versioned artefact
    # with observed values per expectation, not just a pass/fail exit code.


def validate_batch(context, batch_definition, run_id: str):
    validator = context.get_validator(batch_definition=batch_definition,
                                      expectation_suite_name="mrv_activity")
    result = validator.validate(run_id=run_id)
    log.info("validation.ge", success=bool(result.success), run_id=run_id,
             failed=[r.expectation_config.type for r in result.results if not r.success])
    return result
# --- Soda: checks as configuration, evaluated on a schedule ------------------
checks for mrv_activity:
  - missing_count(activity_id) = 0
  - duplicate_count(activity_id) = 0
  - invalid_count(activity_unit) = 0:
      valid values: [MJ, kWh, t, ha]
  - avg(co2e_tonnes) between 0 and 50000:
      name: Mean tonnage within plausible envelope
  # Readable by a non-engineer, and detached from the run that produced the data —
  # this finds the problem after a consumer has already read the table.

The Pandera version fails the task that computed the bad data, which is what a blocking gate must do. The Great Expectations version produces a result document naming every expectation and its observed value, which is what an auditor wants. The Soda version is legible to a domain expert who does not write Python, which is what gets subject-matter knowledge into the checks. Those three virtues are largely orthogonal, which is why mature pipelines frequently run two of them.

A layered arrangement that uses each framework for what it is good at A three-layer arrangement. The innermost layer is Pandera schemas on transformation function boundaries, blocking, running in-process on every task, and carrying the geospatial invariants as dataframe checks. The middle layer is a Great Expectations suite run once per stage over the written artefact, producing the versioned validation document that enters the evidence store. The outer layer is Soda scanning the published warehouse tables on a schedule, owned by analysts and producing trend alerts rather than gates. A panel notes that the completeness check sits outside all three because it compares against an expected set the data cannot know. Layered, not chosen Each framework does one of the three jobs well. Inner · Pandera on every task blocking · in-process · geospatial invariants as dataframe checks Middle · Great Expectations once per stage over the written artefact · produces the versioned evidence document Outer · Soda on published tables analyst-owned · trends and alerts, never a gate Outside all three partition completeness, compared against an expected set derived from the acquisition catalogue the data cannot know it

Compliance Gating & Audit Trail Generation

For a verifier, the framework is invisible and its output is not. What must survive is a per-run record naming every check, its classification as blocking or advisory, the observed value on failure, and the disposition of each failure. Great Expectations produces most of that natively, which is its strongest argument in a compliance setting; Pandera produces rich exceptions that must be captured and serialised deliberately; Soda produces scan results that are excellent for trends and detached from the run that created the data.

Whichever you choose, three things belong in the evidence store rather than in the framework’s own UI. The suite version, because a check-set change alters what “passed” means and is therefore a control change requiring review. The observed values for failures, since “expectation failed” is not evidence while “27 rows exceeded 4.2 tCO₂e/MWh, maximum 118.4, all from SITE-14” is. And the disposition of every failure — corrected, excepted with an owner, or excluded and disclosed — which no framework tracks because it is a workflow rather than a check.

Route those into the evidence stream described under MRV pipeline observability and failure modes, keyed on the same run identifier as the lineage record, so a verifier can move from a figure to its checks without reconciling identifiers by hand.

Production Integration

  1. Write the four hard checks first — units from metadata, geometry and CRS, distributional drift, partition completeness — as plain functions, independent of any framework.
  2. Choose the blocking layer by where you need failure to happen. If it must fail the task that computed the data, that is Pandera. If it must fail at the stage boundary over the written artefact, that is Great Expectations.
  3. Wire the geospatial invariants into the blocking layer’s extension mechanism, keeping them as the plain functions they already are rather than rewriting them as framework primitives.
  4. Add the evidence layer if the blocking layer does not produce a durable document. In practice this means adding Great Expectations alongside Pandera, or serialising Pandera’s failure cases yourself.
  5. Add the analyst layer last, and only if domain experts will actually author checks in it. An unused Soda deployment is operational cost with no return.
  6. Version the suite as a released artefact and record its version on every run, exactly as you version the emission factor tables.

For per-partition overhead, measure before assuming: Pandera’s schema validation over a tile-month dataframe is typically low single-digit milliseconds, while a Great Expectations validation run carries per-invocation setup that matters at ten thousand invocations a night. That asymmetry is the practical reason the layered arrangement puts Pandera inside the loop and Great Expectations at the stage boundary rather than the reverse.

Frequently Asked Questions

Which framework should a small team pick if they can only run one?

Pandera, with the evidence document serialised by hand. It is the only one of the three that blocks at the point of computation with negligible overhead, it expresses custom geospatial checks as ordinary Python, and its schemas live next to the code they guard so they are maintained rather than forgotten. The gap it leaves — a durable, human-readable validation report — is perhaps fifty lines of serialisation, which is far less work than adopting a second framework.

Is Great Expectations worth it purely for the evidence layer?

In a reasonable-assurance setting, frequently yes. The validation result document — every expectation, its parameters, and its observed value, versioned and retained — is close to exactly what a verifier asks for, and reproducing it faithfully is more work than it first appears. In a limited-assurance setting where record-level substantiation is not demanded, the same evidence can be met with a much simpler serialisation and the operational overhead is harder to justify.

Can Soda replace either of the others?

No, because it scans data that has already been published. That is a genuine and useful role — it catches problems that only appear in aggregate, and it lets analysts own checks in a syntax they will actually write — but a monitor is not a gate. A pipeline whose only validation is a scheduled warehouse scan will publish bad data and then find out, which is precisely the failure mode the blocking gates exist to prevent.

How do these frameworks handle geometry columns?

None of them natively, which is the single most important thing to know before choosing. Pandera can validate a GeoDataFrame through custom dataframe checks that call shapely and pyproj directly, which is the least friction of the three. Great Expectations supports custom expectations that can do the same with more ceremony. Soda, operating over warehouse SQL, can reach geometry only through whatever spatial functions the warehouse exposes, which is usually far less than the Python stack offers.

Does validation belong in the orchestrator or in the transformation code?

In the transformation code, invoked by the orchestrator. A validation encoded as a separate orchestrator task can be skipped, reordered, or forgotten when a new pipeline is written, whereas a schema on the function boundary travels with the function. Use the orchestrator to schedule the stage-boundary evidence run, and keep the blocking checks where the data is created — the same principle that keeps methodology logic out of the orchestrator generally.

How should a validation suite be versioned and released?

Like any other control: a version number, a changelog naming what changed and why, and a review before it takes effect. A suite change alters what “passed” means, so a run that passes under version 4 and would have failed under version 3 is materially different from one that passed under both. Record the suite version on every run, keep old versions runnable so a historical period can be re-validated under the rules that applied to it, and time adoption to a period boundary wherever possible.

The corollary is that adding a check is not free. Every new blocking check is a new way for the pipeline to stop, and a check added without an owner and a documented response is a future outage with no runbook. The healthy pattern is to add a check as advisory first, watch its fire rate for a period, and promote it to blocking only once its false-positive behaviour is understood.

What does a validation failure cost in practice?

Less than people fear if the checks are well classified, and considerably more if they are not. A blocking failure on a genuine defect costs one re-run and a source fix; a blocking failure on a legitimate edge case costs an incident, an override, and a small erosion of trust in the gate. That asymmetry is the argument for keeping the blocking set small and absolute: every check in it should describe something that is never legitimate, so a failure is always a real finding rather than a judgement call at two in the morning.