Versioning Emission Factor Databases for Reproducible MRV #

A carbon figure is only reproducible if the emission factor behind it is reproducible, and most pipelines fail that test the first time an upstream table is revised. This guide sits under the MRV Data Schema Reference cluster, where it complements the canonical Parquet schema data dictionary for MRV — the schema defines what a factor row looks like, and this page defines which version of it a computation is allowed to use. It is also a direct dependency of orchestrating MRV data pipelines, because a reproducible backfill is impossible if the factor lookup silently drifts between the original run and the re-run.

The core problem is temporal. When IPCC, DEFRA, or ecoinvent publish a revision, a naive pipeline that joins activity data against “the current factor table” will quietly change last year’s already-reported numbers the next time it recomputes. Emission-factor versioning fixes this by making factor tables immutable and content-addressed, tagging each release with a semantic version, and joining on an effective date so that a 2021 activity always resolves to the 2021-era factor even when the pipeline is executed in 2026. Every output then carries the exact factor_version it consumed, and that value flows into the lineage record an auditor traces — the same provenance discipline described in MRV data lineage and provenance tracking.

Root Cause Analysis #

Emission-factor drift is not a data-quality bug; it is a modelling omission. Three structural causes explain why an otherwise-correct pipeline produces irreproducible numbers.

First, mutable “current-table” lookups conflate two different times. A carbon computation has a transaction time (when the pipeline ran) and a valid time (the period the activity data describes). When a pipeline joins against whatever factor table is live at run time, it collapses those two axes: recomputing a 2021 emission in 2026 silently applies a 2024 factor revision to 2021 activity. IPCC AR revisions, annual DEFRA conversion-factor updates, and ecoinvent’s periodic dataset releases each shift published factors by anywhere from a few percent to double digits, so a recompute can move a previously reported total by more than the uncertainty band the figure was certified against. The number changes, nobody edited any activity record, and no verifier can reconstruct the original.

Second, factor tables are treated as configuration rather than versioned data. A CSV dropped into a shared bucket and overwritten in place has no identity: two runs that both claim to use “the DEFRA factors” may have consumed byte-different files. Without a semantic version and a content hash, there is no way to assert that a re-run used the same inputs, and ISO 14064-3 reproducibility becomes unprovable. The fix is to make each release immutable and content-addressed — the hash is the identity — and to attach a semantic version whose major component signals a methodology-breaking change.

Third, outputs do not record which factor they used. Even a correct as-of join is worthless for audit if the resolved factor_version never lands in the output row. When that column is null, the lineage graph has a hole exactly where a verifier looks first, and the only way to answer “which factor produced this tonne” is to guess from run timestamps. Pinning the version into every row, and gating on the absence of a pin, closes that hole before export.

Diagnostic Pipeline / Pre-Flight Validation #

Before recomputing anything, inspect the existing outputs and the incoming factor release for the two failure signatures above: rows whose factor_version is null (an unpinned join) and rows whose resolved factor would change under a new release (drift). The diagnostic below flags both without mutating state, so a verifier or an orchestration gate can decide whether a recompute is safe.

from __future__ import annotations

import hashlib

import pandas as pd
import pyarrow.parquet as pq
import structlog

log = structlog.get_logger()


def content_hash_parquet(path: str) -> str:
    """Deterministic content address for an immutable factor release."""
    with open(path, "rb") as fh:
        digest = hashlib.sha256(fh.read()).hexdigest()
    return f"sha256:{digest[:16]}"


def diagnose_factor_drift(
    outputs: pd.DataFrame,
    factors_old: pd.DataFrame,
    factors_new: pd.DataFrame,
    release_path: str,
) -> dict:
    """Detect unpinned joins and factor drift before any recompute.

    outputs must carry: factor_id, activity_date, factor_version.
    factor tables carry: factor_id, valid_from, valid_to, factor_value, factor_version.
    """
    issues: list[str] = []

    # 1. Unpinned joins: any output row without a resolved factor_version.
    unpinned = int(outputs["factor_version"].isna().sum())
    if unpinned:
        issues.append(f"unpinned_rows:{unpinned}")

    # 2. Drift: resolve each activity_date against old vs new release and compare.
    def _asof(factors: pd.DataFrame) -> pd.Series:
        merged = pd.merge_asof(
            outputs.sort_values("activity_date"),
            factors.sort_values("valid_from"),
            left_on="activity_date",
            right_on="valid_from",
            by="factor_id",
            direction="backward",
        )
        # A row is only valid if activity_date < valid_to for the matched interval.
        merged.loc[merged["activity_date"] >= merged["valid_to"], "factor_value"] = pd.NA
        return merged["factor_value"].reset_index(drop=True)

    old_vals, new_vals = _asof(factors_old), _asof(factors_new)
    drifted = int((old_vals.fillna(-1) != new_vals.fillna(-1)).sum())
    if drifted:
        issues.append(f"drifted_rows:{drifted}")

    report = {
        "release_hash": content_hash_parquet(release_path),
        "n_outputs": int(len(outputs)),
        "unpinned_rows": unpinned,
        "drifted_rows": drifted,
        "safe_to_recompute": not issues,
        "issues": issues,
    }
    log.info("efdb.preflight", **report)
    return report

A non-empty drifted_rows count is not a failure to suppress — it is the signal that a new factor release genuinely changes historical results, which is exactly the condition an effective-dated join must prevent from leaking into already-reported periods. The diagnostic makes that decision explicit rather than discovering it after publication.

Deterministic Transformation Logic #

The transformation is an as-of (bitemporal) merge: each activity record is matched to the single factor row whose validity interval [valid_from, valid_to) contains the activity_date, keyed by factor_id. This guarantees era-correctness — a 2021 activity binds to the factor that was in force in 2021 regardless of later revisions. The routine below performs the interval join, pins both factor_version and the release content hash into every output row, emits a per-output content hash for downstream lineage, and refuses to return an unpinned frame.

from __future__ import annotations

import hashlib

import pandas as pd
import pyarrow as pa
import structlog

log = structlog.get_logger()


def join_effective_dated_factors(
    activity: pd.DataFrame,
    factors: pd.DataFrame,
    release_hash: str,
) -> pd.DataFrame:
    """As-of join activity data to era-correct emission factors, pinning version.

    activity: activity_id, activity_date (datetime64), factor_id, activity_qty
    factors:  factor_id, valid_from, valid_to (datetime64), factor_value,
              factor_version, unit
    """
    if activity["activity_date"].isna().any():
        raise ValueError("activity_date contains nulls; cannot resolve era factor.")

    left = activity.sort_values("activity_date")
    right = factors.sort_values("valid_from")

    # merge_asof(direction="backward") picks the latest valid_from <= activity_date
    joined = pd.merge_asof(
        left, right,
        left_on="activity_date", right_on="valid_from",
        by="factor_id", direction="backward",
    )

    # Interval upper-bound check: reject matches where the interval has expired.
    expired = joined["activity_date"] >= joined["valid_to"]
    joined.loc[expired, ["factor_value", "factor_version"]] = pd.NA

    # Validation gate: every row must resolve to a pinned, in-force factor.
    unresolved = int(joined["factor_version"].isna().sum())
    if unresolved:
        raise ValueError(
            f"{unresolved} activity rows have no in-force factor; "
            "extend factor validity intervals or fix activity_date."
        )

    joined["emission"] = joined["activity_qty"] * joined["factor_value"]
    joined["factor_release_hash"] = release_hash

    # Content hash of the pinned result — the addressable identity of this output.
    canonical = joined[
        ["activity_id", "activity_date", "factor_id",
         "factor_version", "emission"]
    ].to_csv(index=False).encode("utf-8")
    joined.attrs["output_hash"] = f"sha256:{hashlib.sha256(canonical).hexdigest()[:16]}"

    log.info(
        "efdb.joined",
        n_rows=int(len(joined)),
        factor_versions=sorted(joined["factor_version"].dropna().unique().tolist()),
        factor_release_hash=release_hash,
        output_hash=joined.attrs["output_hash"],
    )
    return joined

The join is deterministic in both temporal axes: direction="backward" fixes the valid-time resolution, and the pinned factor_release_hash fixes the transaction-time inputs. Re-running the same activity data against the same release hash reproduces the same output_hash byte-for-byte, which is what makes the output replayable years later. A small effective-dated factor table makes the era resolution concrete:

An activity dated 2021-06-14 against defra_grid_uk resolves to defra-2021.1.0 (0.21233) every time, even when the join is executed in 2026 with three later releases loaded.

Compliance Gating & Audit Trail Generation #

The pinned columns are the audit artifact. factor_version, factor_release_hash, and the per-output output_hash together let a third-party verifier answer the two questions ISO 14064-3 reproducibility turns on: which factor produced this figure, and can the figure be regenerated identically. Because the factor table is immutable and content-addressed, the verifier can fetch the exact release by hash, re-run the effective-dated join, and confirm the output_hash matches — no timestamp guesswork.

Key gates to enforce on every run, including passing ones:

1. No unpinned rows. The join raises if any factor_version is null; a downstream schema check on the Parquet output should re-assert non-null on that column so an unpinned frame can never be published.
2. Immutable release identity. Reject any factor load whose content hash matches an existing version tag but whose bytes differ, or whose bytes match a version already published under a different tag — either is a versioning error, not a data update.
3. Semantic version discipline. A major-version bump (e.g. ipcc-2019.x to ipcc-2024.1.0) signals a methodology change and must open a new validity interval rather than overwriting an existing one, so historical periods keep resolving to their era factor. This is the boundary that stops IPCC and DEFRA revisions from silently rewriting the past.
4. Drift disclosure. When diagnose_factor_drift reports a non-zero drifted_rows, the count and the two release hashes are recorded, evidencing that the new factor was applied only to periods it is in force for — the transparency CSRD ESRS E1 expects around restated figures.

These records flow into the lineage graph exactly as the canonical Parquet schema prescribes, where factor_version becomes a queryable provenance edge rather than a buried attribute.

Production Integration #

Deploy the routine inside the orchestrator described in orchestrating MRV data pipelines, following a fixed ingest → diagnose → transform → validate → export → submit sequence:

1. Ingest. Load the incoming factor release as an immutable Parquet object, compute its content hash with content_hash_parquet, and register it under its semantic version — refusing the load if that version already exists with different bytes.
2. Diagnose. Run diagnose_factor_drift against the live outputs to surface any unpinned rows and to quantify how many historical results a new release would move, so the orchestrator can decide whether a recompute touches only in-force periods.
3. Transform. Call join_effective_dated_factors with the release hash, resolving each activity to its era-correct factor and pinning factor_version, factor_release_hash, and the per-output output_hash.
4. Validate. Re-assert non-null factor_version on the output frame and confirm the recomputed output_hash matches the prior run for any unchanged period — the reproducibility check an idempotent backfill depends on.
5. Export. Serialize to Parquet with the pinned columns intact, so the addressable factor identity travels with the emission figures.
6. Submit. Forward factor_version and both hashes into the lineage record consumed by MRV data lineage and provenance tracking, and feed the resolved factor values into downstream stages such as emission factor uncertainty mapping, which propagates each pinned factor’s variance.

By making factor tables immutable and content-addressed, joining on effective date, and pinning the resolved version into every output and its lineage, the pipeline guarantees that a figure reported in 2021 stays byte-identical when regenerated in 2026 — the reproducibility contract that lets automated MRV survive third-party verification.

Related guides #

• MRV Data Schema Reference — the parent cluster defining the schema these pinned factors populate.
• The Canonical Parquet Schema: A Data Dictionary for MRV — the field-level contract for the factor_version and hash columns.
• Orchestrating MRV Data Pipelines — the orchestration layer whose idempotent recomputes rely on pinned factors.
• Emission Factor Uncertainty Mapping — the downstream stage that propagates the variance of each resolved factor.
• MRV Data Lineage & Provenance Tracking — where the pinned factor_version becomes an auditable provenance edge.