Data Governance: The Platform Foundation
Every Feature Has a Lineage

Custom JSON schema validation (Python jsonschema or Pydantic) was the initial design. It was replaced with TFX ExampleValidator for four reasons. (1) TFX is stateful — it learns statistics from observed data and can flag distributional drift, not just structural violations. A temperature reading within the valid range but 10 standard deviations above fleet mean would pass JSON schema validation and fail TFX's anomaly detection. (2) TFX integrates natively with Vertex AI Pipelines — the same schema definitions used in offline training are enforced at serving time, eliminating train-serve skew. (3) TFX's Schema artifact is versionable in the ML Metadata store — schema evolution is tracked automatically. (4) TFX Validate data quality expectations are living documentation satisfying EU AI Act Article 11's training data requirement. Custom JSON schema has none of these properties.

The simpler design — reject schema-violating records at the Pub/Sub subscriber, log the error, and discard — was rejected for two reasons. (1) Data loss is irreversible in a real-time sensor system. A regional system sending 847 records per batch cannot easily re-send data from 4 hours ago. Quarantine preserves the raw record in BigQuery, allowing reprocessing once the schema mapping is confirmed; reject-on-arrival means the data is permanently gone. (2) Schema violations often indicate a legitimate upstream system update, not corrupted data. The APAC-East firmware update is the canonical example — the data is valid, only the schema registration is missing. Reject-on-arrival silently discards valid, irreplaceable telemetry. Quarantine-then-review surfaces the issue to a human who can determine whether the data is recoverable.

ADR-AQ02 was documented in the Asset IQ module as the schema standardisation decision. It is cross-referenced here because Data Governance is the component that makes ADR-AQ02 enforceable at runtime. The unified Pub/Sub canonical schema is not a convention that source systems are expected to follow — it is a gate that every record must pass. Without Data Governance, ADR-AQ02 is a design aspiration; with Data Governance, it is a hard technical constraint. Any regional system that updates its schema without registering the change will have its records quarantined until a steward reviews them. This creates a virtuous feedback loop — schema violations are visible, immediate, and operationally costly (P2 alert), incentivising regional teams to register schema changes before deploying.

Two PII handling approaches were evaluated. (1) Reject-on-PII-detection: discard records containing PII in free_text_notes. Rejected because field engineer notes are operationally valuable — they often contain maintenance observations that inform the RUL model's context features, and silent discard removes recoverable signal. (2) DLP Hold with redact-then-reinstate: quarantine the raw record, alert the data steward with the specific detected entity types (e.g. PERSON_NAME, EMAIL_ADDRESS), and require an explicit redaction action before reinstatement. The steward does not see a "reinstate as-is" option for DLP holds — only "redact detected fields + reinstate" or "discard".

This distinction is critical: schema quarantine reinstatement is a mapping approval; DLP reinstatement without redaction would write raw PII to the Feature Store, violating GDPR data minimisation (Art. 5(1)(c)). The DLP Hold state is therefore architecturally distinct from the QUARANTINED state — different steward UI, different approval action, different audit log entry (ae_governance.dlp_remediation_log vs ae_governance.discard_log). Cloud DLP is scoped to free_text_notes only; structured numeric/timestamp fields are not inspected. Cost: ~$162/month at ClaraVis volume — see §2.1.