MRV Data Schema Reference #

The MRV Data Schema Reference defines the shared data-contract layer that every stage of a measurement, reporting, and verification platform reads from and writes to — the single canonical Parquet schema that turns a loose collection of rasters, plot tables, and emission-factor lookups into an addressable, typed, versioned data product. It is the connective tissue of the Pipeline Orchestration & Compliance Reference stack: where orchestrating MRV data pipelines decides when a task runs and in what order, the schema decides what shape the data crossing every task boundary must take, and it does so with the same rigour the platform applies when it maps outputs to carbon registry standards and methodologies. Without an enforced contract, an orchestration DAG is just a way to move malformed data around faster.

This reference treats the schema as the primary compliance artifact it is. A carbon figure that reaches a registry carries an implicit claim: that every tonne was measured in known units, projected in a declared coordinate reference system, and joined against a named version of an emission-factor table. The schema is where those claims become machine-checkable rather than narrated in a methodology PDF. The two child guides below drill into the specifics — the full column-by-column canonical Parquet schema data dictionary for MRV and the discipline of versioning emission-factor databases for reproducible MRV — while this page establishes the contract model, the enforcement pattern, and the failure modes that make an unenforced schema so dangerous.

Role in the MRV Workflow #

The schema reference sits at the boundary between every producing stage and the canonical store that the rest of the platform reads. In an MRV pipeline, data arrives from radically different worlds: reflectance and backscatter rasters reduced to per-parcel carbon-density tables, field crews uploading plot inventories, and third-party emission-factor databases published on their own release cadence. Left unmediated, each producer imposes its own column names, its own units, its own idea of what a coordinate means. The contract layer collapses that heterogeneity into one shape, so that a downstream consumer — an uncertainty pass, a registry submission builder, an auditor’s query — never has to ask which flavour of emissions column it is looking at.

Two enforcement philosophies compete here, and the choice is load-bearing. Schema-on-read defers validation to query time: data is written in whatever shape it arrives, and each consumer interprets it. That is expedient for exploratory analytics but corrosive for MRV, because it means the same raw file can yield two different tonnage figures depending on who reads it and with what assumptions. Schema-on-write — the model this reference mandates — validates at ingestion, refuses non-conformant records at the gate, and guarantees that anything in the canonical store already satisfies the contract. The cost is a stricter ingestion path; the payoff is that the system of record is trustworthy by construction, and an auditor can reason about a single, enforced shape rather than a distribution of possible interpretations.

The physical substrate for that contract is columnar Parquet, extended with GeoParquet for the spatial columns. Parquet earns its place because its footer carries a typed schema that travels with the data: dtypes, nullability, and arbitrary key–value metadata are embedded in every file, so the contract is not a separate document that can drift out of sync with the bytes it describes. GeoParquet adds the piece MRV cannot live without — a standardized geo metadata block that records the geometry encoding and, critically, the coordinate reference system as embedded PROJJSON. A geometry column without a declared CRS is not data, it is a guess; the schema forbids that guess from ever entering the store.

Every column in the canonical schema carries three things beyond its name: a physical dtype, a declared unit, and a human-readable description. Units in particular are promoted from convention to contract, because in carbon accounting a silent factor-of-a-thousand error between kilogrammes and tonnes is not a rounding problem — it is a credit-issuance problem. The table below sketches the core columns at the level a reference resource should; the exhaustive dictionary, including nullability rules, enumerations, and per-column validation predicates, lives in the canonical Parquet schema data dictionary.

Two design decisions in that table deserve emphasis. First, emissions is contractually tCO2e — the unit is part of the column’s identity, not a note in a spreadsheet — so any producer emitting kilogrammes must convert before the gate, and the gate is entitled to range-check magnitudes accordingly. Second, both ef_version and schema_version are first-class columns rather than pipeline-run metadata, because reproducibility demands that a row remember not just its values but the contracts it was computed under. A figure without those two versions is unauditable no matter how precise it looks.

Core Failure Modes #

Three failure modes dominate production MRV schema management. Each is silent by default — the pipeline keeps running and the numbers keep flowing — which is exactly what makes them dangerous. Each has a distinct root cause and a measurable impact on the credibility of the reported figures.

1. Schema drift silently dropping columns. The root cause is permissive I/O: a pandas.read_parquet followed by a to_parquet will happily round-trip whatever columns happen to be present, and a producer that stops emitting uncertainty_rel — because an upstream library changed a default, or a join quietly failed — produces a file that is structurally valid Parquet but semantically incomplete. Under schema-on-read, nothing complains; the column simply vanishes, and every downstream consumer treats its absence as “no uncertainty” rather than “uncertainty unknown”. Observed impact: in a portfolio where the uncertainty column drives conservative deductions, a dropped column removes the deduction entirely, inflating claimable tonnage by the full deduction margin — routinely 8–15% of issuance — until a verifier notices the figures are suspiciously tight. The fix is schema-on-write with an explicit required-column set; drift becomes a hard rejection at the gate rather than a discovery in an audit.

2. Unit ambiguity between tCO2e and kgCO2e. The root cause is treating units as documentation instead of contract. A double column named emissions carries no intrinsic unit, so a producer that switches from tonnes to kilogrammes — or a data source that was always in kilogrammes and was assumed to be in tonnes — introduces a factor-of-1000 error that passes every type check because the dtype never changed. Observed impact is catastrophic and asymmetric: a kg-as-tonnes error overstates emissions by 1000×, which in a removals project could zero out an entire vintage; the reverse understates by 1000×, quietly under-reporting a facility’s footprint against CSRD ESRS E1 disclosure. The fix is to bind the unit to the column in schema metadata and enforce a plausibility range at the gate, so a value three orders of magnitude outside the expected envelope raises rather than reconciles.

3. Unversioned emission-factor joins making runs irreproducible. The root cause is joining against a mutable emission-factor table — a database view, a “latest” CSV, an API endpoint that returns whatever is current — without recording which release was used. The run succeeds, the numbers look fine, and six months later the same pipeline over the same inputs produces different figures because the factor table was revised in the interim. Observed impact: the reporting period can no longer be reconstructed bit-for-bit, which fails the reproducibility requirement of ISO 14064-3 outright; a verifier who reruns the pipeline gets a number that disagrees with the submitted one and has no way to tell whether the discrepancy is an error or an unrecorded factor update. The fix is to pin every join to an immutable, semantically versioned factor release and to persist that ef_version in the row itself — the discipline detailed in versioning emission-factor databases for reproducible MRV.

Deterministic Implementation Architecture #

The enforcement point is a single contract-validation function invoked at every pipeline boundary that writes to the canonical store. It validates dtypes against an explicit pyarrow schema, asserts the declared CRS on the GeoParquet metadata, range-checks the unit-bearing columns to catch magnitude errors, and requires the version columns to be present and non-null. Anything that fails is raised — never coerced, never silently dropped — and the rejection is logged as a structured event so the quarantine record carries a machine-readable reason. The function below uses pyarrow for the authoritative type check and pandera for the column-level predicate layer, with structlog emitting audit-ready JSON telemetry on every decision.

from __future__ import annotations

import re
from datetime import datetime, timezone

import pyarrow as pa
import pyarrow.parquet as pq
import pandera.pandas as pandera
from pandera import Column, Check
import structlog

structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.JSONRenderer(),
    ]
)
log = structlog.get_logger()


class SchemaContractError(Exception):
    """Raised when a record violates the canonical MRV data contract."""


# Authoritative physical schema — the Parquet footer must match this exactly.
CANONICAL_ARROW_SCHEMA = pa.schema([
    ("parcel_id", pa.string()),
    ("geometry", pa.binary()),
    ("reporting_period", pa.date32()),
    ("carbon_stock", pa.float64()),      # unit: tC
    ("emissions", pa.float64()),         # unit: tCO2e — NOT kgCO2e
    ("emission_factor_id", pa.string()),
    ("ef_version", pa.string()),
    ("uncertainty_rel", pa.float64()),
    ("crs_epsg", pa.int32()),
    ("schema_version", pa.string()),
])

# Equal-area CRS is mandatory so area-weighted carbon maths stays honest.
ALLOWED_EPSG = {6933, 3035}          # global equal-area, EU equal-area
SEMVER = re.compile(r"^\d+\.\d+\.\d+$")
# Plausibility envelope for tonnes CO2e per parcel-period; a kg value would blow past it.
MAX_PLAUSIBLE_TCO2E = 5_000_000.0

# Column-level predicate layer: units, ranges, versions, nullability.
CONTRACT = pandera.DataFrameSchema(
    {
        "parcel_id": Column(str, nullable=False, unique=False),
        "carbon_stock": Column(float, Check.ge(0), nullable=False),
        "emissions": Column(
            float,
            Check.in_range(-MAX_PLAUSIBLE_TCO2E, MAX_PLAUSIBLE_TCO2E),
            nullable=False,
        ),
        "emission_factor_id": Column(str, nullable=False),
        "ef_version": Column(str, Check.str_matches(SEMVER), nullable=False),
        "uncertainty_rel": Column(float, Check.in_range(0.0, 1.0), nullable=False),
        "crs_epsg": Column(int, Check.isin(sorted(ALLOWED_EPSG)), nullable=False),
        "schema_version": Column(str, Check.str_matches(SEMVER), nullable=False),
    },
    strict=True,   # reject any unexpected OR missing column — this catches schema drift
    coerce=False,  # never silently cast; a wrong dtype is a violation, not a fixup
)


def _assert_arrow_types(table: pa.Table, run_id: str) -> None:
    """Fail if the Parquet footer schema deviates from the canonical contract."""
    got = {f.name: str(f.type) for f in table.schema}
    want = {f.name: str(f.type) for f in CANONICAL_ARROW_SCHEMA}
    missing = set(want) - set(got)
    extra = set(got) - set(want)
    mistyped = {n: (got[n], want[n]) for n in want & got.keys() if got[n] != want[n]}
    if missing or extra or mistyped:
        log.error("schema.contract.type_mismatch", run_id=run_id,
                  missing=sorted(missing), extra=sorted(extra), mistyped=mistyped)
        raise SchemaContractError(
            f"type contract violated: missing={missing} extra={extra} mistyped={mistyped}")


def _assert_crs(table: pa.Table, run_id: str) -> int:
    """Assert the GeoParquet 'geo' metadata declares an allowed equal-area CRS."""
    import json
    meta = table.schema.metadata or {}
    geo_raw = meta.get(b"geo")
    if geo_raw is None:
        log.error("schema.contract.crs_missing", run_id=run_id)
        raise SchemaContractError("GeoParquet 'geo' metadata absent; CRS undeclared.")
    geo = json.loads(geo_raw.decode("utf-8"))
    primary = geo["primary_column"]
    projjson = geo["columns"][primary].get("crs")
    epsg = None
    if isinstance(projjson, dict):
        epsg = projjson.get("id", {}).get("code")
    if epsg not in ALLOWED_EPSG:
        log.error("schema.contract.crs_rejected", run_id=run_id, epsg=epsg,
                  allowed=sorted(ALLOWED_EPSG))
        raise SchemaContractError(f"CRS EPSG:{epsg} not in equal-area allow-list.")
    return int(epsg)


def validate_canonical_write(path: str, *, run_id: str) -> pa.Table:
    """Contract gate: validate a Parquet file before it enters the canonical store.

    Raises SchemaContractError on any dtype, unit, CRS, or version violation so a
    non-conformant record is quarantined rather than written to the system of record.
    """
    table = pq.read_table(path)
    _assert_arrow_types(table, run_id)
    epsg = _assert_crs(table, run_id)

    df = table.drop(["geometry"]).to_pandas()
    # Cross-check the redundant queryable CRS column against the geo metadata.
    if not (df["crs_epsg"] == epsg).all():
        log.error("schema.contract.crs_column_mismatch", run_id=run_id, geo_epsg=epsg)
        raise SchemaContractError("crs_epsg column disagrees with GeoParquet metadata.")

    try:
        CONTRACT.validate(df, lazy=True)
    except pandera.errors.SchemaErrors as exc:
        failures = exc.failure_cases[["column", "check", "failure_case"]].to_dict("records")
        log.error("schema.contract.predicate_failed", run_id=run_id, failures=failures[:20])
        raise SchemaContractError(f"{len(failures)} predicate violation(s) at the gate.")

    log.info("schema.contract.passed", run_id=run_id, rows=table.num_rows,
             crs_epsg=epsg, schema_version=str(df["schema_version"].iloc[0]),
             validated_at=datetime.now(timezone.utc).isoformat())
    return table

Two properties of this gate are non-negotiable. First, strict=True combined with coerce=False means the contract catches drift in both directions — a dropped required column and an unexpected extra column both raise, and no value is ever silently cast to make a bad file pass. That single pair of flags neutralises the first failure mode. Second, the unit check on emissions is a plausibility envelope, not a mere non-null test: a value that has been mistakenly multiplied by 1000 will breach MAX_PLAUSIBLE_TCO2E and raise, converting the second failure mode from a silent thousand-fold error into a loud rejection. The version columns are validated as strict semver, so a run that failed to pin its ef_version cannot reach the store at all.

Validation, Debugging & Compliance Mapping #

Each design decision in the gate maps to a specific regulatory control, which is what makes the schema a submission artifact rather than an engineering convenience. The dtype contract and required-column set answer ISO 14064-3, whose validation and verification clauses expect the reported data to be complete and consistent across the reporting boundary; a strict schema is the mechanism by which “complete” becomes machine-checkable rather than asserted. The version columns answer the same standard’s reproducibility expectation directly — a verifier who reruns the pipeline against the pinned ef_version and schema_version must obtain the identical figure, and the persisted versions are what make that rerun deterministic. The unit contract feeds the conservativeness principle running through the Verra VM-series methodologies: a magnitude gate that refuses implausible tonnage prevents both the overstatement that would inflate issuance and the understatement that would strand legitimate credit. The declared-CRS assertion underwrites every area-weighted figure, because a tonne-per-hectare number computed in a non-equal-area projection is quantitatively wrong before any accounting rule is applied.

Mapping the gate to disclosure obligations is equally direct. CSRD ESRS E1 demands auditable, traceable climate metrics with explicit data-quality treatment; a canonical store where every row carries its schema version, its emission-factor version, and its declared uncertainty is precisely the traceable substrate an assurance provider needs, and the quarantine stream provides the documented evidence that non-conformant data was excluded rather than quietly absorbed. The dead-letter records themselves are a compliance asset: each carries a structured rejection reason, so an auditor can see not only what entered the store but what was kept out and why.

For debugging, treat three signals as monitored on every run, including the runs that pass. The quarantine rate is the leading indicator of upstream drift — a producer that begins emitting a renamed or retyped column shows up as a sudden spike in gate rejections long before any figure is wrong downstream. The distribution of emissions magnitudes surfaces creeping unit errors that stay just inside the plausibility envelope; a producer that switches units on a low-emissions parcel may not breach the hard gate, but the distribution will shift by orders of magnitude and a percentile monitor will catch it. The set of distinct ef_version values per reporting period reveals unpinned joins: a period that should reference a single frozen factor release but shows several versions is evidence that some path bypassed the pin. Contract tests belong in continuous integration as well as at runtime — a pytest suite that writes deliberately malformed fixtures (a dropped column, a kg-scaled value, an undeclared CRS, an unpinned version) and asserts that validate_canonical_write raises on each is what stops a refactor from silently loosening the contract. That test suite is the schema’s own regression harness, and it is inseparable from the data lineage and provenance tracking layer that records which contract version every archived figure was validated against.

Conclusion #

The MRV Data Schema Reference is the contract that makes an orchestrated carbon pipeline auditable rather than merely automated. By committing to a single canonical Parquet schema, validating on write instead of on read, embedding CRS and units as enforced metadata rather than convention, and versioning both the schema and the emission-factor tables it joins against, an engineering team converts three of the most common silent MRV failures — dropped columns, unit ambiguity, and irreproducible joins — into loud, quarantined rejections that never reach a registry. The gate shown here is deliberately unforgiving because the alternative is a system of record whose figures cannot be defended under third-party verification. From here, the two child guides go deeper: the full canonical Parquet schema data dictionary specifies every column and its predicates, and versioning emission-factor databases for reproducible MRV makes the pinned-join discipline concrete.

Related guides #

• Pipeline Orchestration & Compliance Reference — the parent stack this data-contract layer anchors.
• The Canonical Parquet Schema: A Data Dictionary for MRV — the exhaustive column-by-column dictionary this reference summarizes.
• Versioning Emission Factor Databases for Reproducible MRV — pinning factor releases so runs reproduce bit-for-bit.
• Orchestrating MRV Data Pipelines — the scheduling layer that invokes the contract gate at every boundary.
• MRV Data Lineage & Provenance Tracking — the evidence layer that records which contract version every figure was validated against.