The Canonical Parquet Schema: A Data Dictionary for MRV #

Every measurement, reporting, and verification pipeline eventually converges on one physical artifact: the table an auditor actually reads. This page is the authoritative column-by-column dictionary for that table — the canonical GeoParquet dataset referenced across the MRV Data Schema Reference discipline within the wider Pipeline Orchestration & Compliance Reference stack. It pins down the exact Arrow dtype, unit, nullability, and semantic contract for each field so that activity data, geometry, emission factors, computed CO2e, uncertainty, and lineage all survive a third-party verification without a single “what does this column mean?” clarifying email.

A schema is not documentation you write after the fact; it is a runtime contract you enforce on every write. The factors that populate emission_factor and factor_version are themselves versioned artifacts governed by versioning emission factor databases for reproducible MRV, and the identifiers that stitch this table back to its inputs are the raw material for MRV data lineage and provenance tracking. Treating the schema as canonical means one physical layout, one set of units, and one validation gate stand between an upstream transform and the registry — no per-project dialects, no silently-coerced columns, no ambiguity about whether co2e_tonnes is metric tonnes or short tons.

Why a Single Canonical Schema Exists #

The alternative to a canonical schema is a proliferation of near-identical dialects, and every dialect is a place a verification stalls. Three failure patterns justify the discipline.

First, unit ambiguity is silent and expensive. A column named emissions with no declared unit is worthless to an auditor and dangerous to a downstream aggregation. Is it kilograms or tonnes? CO2 or CO2e? Per record or per hectare? When two feeds disagree by a factor of 1000 because one used kilograms, the error does not raise — it produces a plausible-looking total that fails reconciliation weeks later. The canonical schema fixes co2e_tonnes as metric tonnes of CO2 equivalent, full stop, and pairs activity_value with an explicit activity_unit string so the factor’s denominator is never inferred.

Second, nullability is a contract, not an accident. If lineage_hash is nullable, a record can reach the registry with no traceable provenance, and no gate will have stopped it. Declaring co2e_tonnes, crs_epsg, and lineage_hash as non-nullable turns “we forgot to compute it” into a write-time failure rather than an audit-time finding. The dtype system does the enforcement the code review would otherwise have to.

Third, schema drift breaks reproducibility. Parquet stores its schema in the file footer, so a column that quietly changes from int32 to int64, or a factor_version that migrates from string to dictionary encoding, produces files that a strict reader rejects and a lenient reader silently coerces. Both outcomes are unacceptable under ISO 14064-3, which requires that the same inputs regenerate the same outputs. Pinning the Arrow schema — dtypes, field order, and metadata — makes drift a caught exception.

The Core Column Dictionary #

The table below is the authoritative definition of the canonical mrv_emissions dataset. Every field lists its Arrow (pyarrow) type, physical unit, whether nulls are permitted, and its semantic contract. Column families follow the SVG grouping: identity, spatial, activity, factor, result, uncertainty, and lineage.

Two conventions in this table are load-bearing. Timestamps are always microsecond-precision UTC-tagged — a naive local timestamp is a rejection, because reporting-period boundaries decide which credits fall in which vintage. And the emission_factor sign convention (negative for removals, positive for emissions) means co2e_tonnes can be summed directly across a project without branching on sequestration versus emission.

Partitioning, Metadata & Storage Conventions #

Physical layout is part of the schema. The dataset is partitioned Hive-style on three keys chosen to match the dominant query patterns — a verifier pulls one project’s one reporting year under one factor version — so predicate pushdown prunes whole directories before a single row group is read. File-level GeoParquet metadata travels in each file’s footer so any Parquet reader can recover the geometry column and CRS without an external sidecar.

Partition keys are folder-encoded and must not be duplicated as row columns — storing factor_version twice invites the folder and the column to disagree after a careless rewrite. The reader reconstructs them from the path.

The pyarrow Schema and Validation Gate #

The schema below is the executable form of the dictionary. validate() is the gate every write passes through: it checks field presence, exact dtype, unit metadata, CRS, and the non-null contract, logs a structured event, and raises on any mismatch rather than coercing. Units and semantics ride in per-field Arrow metadata so they are serialized into the Parquet footer and recoverable by any consumer.

from __future__ import annotations

import pyarrow as pa
import pyarrow.parquet as pq
import structlog

log = structlog.get_logger()

# Per-field metadata carries the unit contract into the Parquet footer.
def _f(name: str, dtype: pa.DataType, unit: str, nullable: bool) -> pa.Field:
    return pa.field(name, dtype, nullable=nullable, metadata={b"unit": unit.encode()})

CANONICAL_MRV_SCHEMA = pa.schema([
    # identity
    _f("record_id", pa.string(), "uuidv7", False),
    _f("project_id", pa.string(), "registry_id", False),
    # spatial
    _f("geometry", pa.binary(), "wkb", False),
    _f("crs_epsg", pa.int32(), "epsg", False),
    _f("area_ha", pa.float64(), "ha", False),
    # activity
    _f("period_start", pa.timestamp("us", tz="UTC"), "utc", False),
    _f("period_end", pa.timestamp("us", tz="UTC"), "utc", False),
    _f("activity_value", pa.float64(), "activity_unit", False),
    _f("activity_unit", pa.string(), "ucum", False),
    # factor
    _f("factor_id", pa.string(), "factor_key", False),
    _f("factor_version", pa.string(), "semver", False),
    _f("emission_factor", pa.float64(), "tco2e_per_activity_unit", False),
    # result
    _f("co2e_tonnes", pa.float64(), "tco2e", False),
    # uncertainty
    _f("uncertainty_pct", pa.float64(), "percent", True),
    _f("tier", pa.int8(), "ipcc_tier", False),
    _f("method_id", pa.string(), "methodology_key", False),
    # lineage
    _f("lineage_hash", pa.string(), "sha256_hex", False),
    _f("ingested_at", pa.timestamp("us", tz="UTC"), "utc", False),
])

VALID_CRS_EPSG = {4326, 6933}  # WGS84 for storage, EQ-Earth (6933) for area math


class SchemaViolation(ValueError):
    """Raised when a table breaches the canonical MRV contract."""


def validate(table: pa.Table, *, strict_crs: bool = True) -> pa.Table:
    """Enforce the canonical MRV schema; raise on any structural mismatch."""
    issues: list[str] = []

    # 1. Field presence, order, dtype and nullability must match exactly.
    got = {f.name: f for f in table.schema}
    for expected in CANONICAL_MRV_SCHEMA:
        actual = got.get(expected.name)
        if actual is None:
            issues.append(f"missing_column:{expected.name}")
            continue
        if actual.type != expected.type:
            issues.append(f"dtype:{expected.name}:{actual.type}!={expected.type}")
        if actual.nullable and not expected.nullable:
            issues.append(f"nullability:{expected.name}:declared_nullable")

    # 2. Non-nullable columns must contain no nulls in the data itself.
    for expected in CANONICAL_MRV_SCHEMA:
        if not expected.nullable and expected.name in table.column_names:
            if table.column(expected.name).null_count > 0:
                issues.append(f"null_values:{expected.name}")

    # 3. CRS gate: every geometry must carry a whitelisted EPSG code.
    if strict_crs and "crs_epsg" in table.column_names:
        codes = set(table.column("crs_epsg").drop_null().unique().to_pylist())
        if not codes <= VALID_CRS_EPSG:
            issues.append(f"crs:{codes - VALID_CRS_EPSG}_not_in_{VALID_CRS_EPSG}")

    # 4. Unit contract: co2e_tonnes must be tagged tCO2e, never coerced.
    ef = table.schema.field("co2e_tonnes") if "co2e_tonnes" in table.column_names else None
    if ef is not None and (ef.metadata or {}).get(b"unit") != b"tco2e":
        issues.append("unit:co2e_tonnes_not_tagged_tco2e")

    if issues:
        log.error("mrv.schema.rejected", rows=table.num_rows, issues=issues)
        raise SchemaViolation("; ".join(issues))

    log.info("mrv.schema.validated", rows=table.num_rows,
             columns=table.num_columns, crs=sorted(codes) if strict_crs else None)
    return table.cast(CANONICAL_MRV_SCHEMA)


# GeoParquet file-level metadata written into the footer on export.
GEO_METADATA = {
    b"geo": (
        b'{"version":"1.1.0","primary_column":"geometry",'
        b'"columns":{"geometry":{"encoding":"WKB",'
        b'"geometry_types":["Polygon","MultiPolygon"],'
        b'"crs":"EPSG:4326","bbox":[-73.99,-33.75,-34.79,5.27]}}}'
    )
}


def write_partition(table: pa.Table, root: str) -> None:
    """Validate, then write a Zstd GeoParquet dataset partitioned Hive-style."""
    table = validate(table)
    table = table.replace_schema_metadata({**(table.schema.metadata or {}), **GEO_METADATA})
    pq.write_to_dataset(
        table, root_path=root,
        partition_cols=["country_iso", "reporting_year", "factor_version"],
        compression="zstd", compression_level=9,
        row_group_size=1_000_000,
    )
    log.info("mrv.partition.written", root=root, rows=table.num_rows)

The cast at the end of validate() is deliberate: once the structural checks pass, casting to the canonical schema pins dtypes and field order so no downstream writer can reintroduce drift. Note that country_iso and reporting_year are supplied as partition columns to write_to_dataset and encoded into the path — they are not part of the row-level schema above, which keeps the folder and the footer from ever disagreeing.

Mapping Columns to Compliance Evidence #

Each column family answers a specific evidentiary need. Verifiers under ISO 14064-3 do not read prose; they trace a reported figure back to the record that produced it, and the canonical schema is designed so that trace is a single query. The mapping below is what makes the dataset a submission artifact.

The uncertainty_pct column deserves emphasis. CSRD ESRS E1 expects disclosures to state estimation uncertainty rather than a single unqualified number, and the Verra VM-series ties a quantified deduction to that same value — the confidence envelopes produced by emission factor uncertainty mapping land in this column and flow straight into the deduction rule with no manual re-keying.

Production Integration #

The schema sits at the join between transformation and submission, enforced on write. In an orchestrated pipeline the canonical table is produced and validated in a fixed ingest → diagnose → transform → validate → export → submit sequence:

1. Ingest. Read upstream activity-data feeds and the resolved factor set, pinning the exact factor_version so the run is reproducible.
2. Diagnose. Confirm every input geometry carries a machine-readable CRS and that monitoring-period boundaries are UTC-tagged and non-overlapping before any arithmetic.
3. Transform. Compute co2e_tonnes and attach uncertainty_pct, tier, and method_id, then derive lineage_hash over the input artifact hashes and code version.
4. Validate. Call validate() — a SchemaViolation fails the task loudly rather than shipping a coerced table; the structured log event is the first artifact an auditor sees.
5. Export. Write with write_partition() so the Zstd GeoParquet lands under country_iso/reporting_year/factor_version with GeoParquet metadata in every footer.
6. Submit. Hand the partitioned dataset to registry integration, where record_id and lineage_hash become the queryable spine of the audit trail.

Enforced this way, the canonical schema stops being a document that drifts from reality and becomes the contract the pipeline cannot violate silently. Every figure that reaches a registry ships with its geometry, its factor version, its uncertainty, and a hash that reconstructs how it was produced — which is precisely what turns a Parquet file into admissible carbon accounting evidence.

Related guides #

• MRV Data Schema Reference — the parent discipline defining the canonical data contracts this dictionary specifies.
• Versioning Emission Factor Databases for Reproducible MRV — how the factor_id / factor_version fields stay reproducible across re-baselines.
• Pipeline Orchestration & Compliance Reference — the wider stack that produces and submits this dataset.
• MRV Data Lineage & Provenance Tracking — the provenance layer the lineage_hash column anchors.
• Emission Factor Uncertainty Mapping — the source of the uncertainty_pct envelopes this schema carries.