Every data team has a quality dashboard. Red and green indicators. Freshness checks. Row count comparisons. The dashboard says green. Then a downstream report serves wrong numbers to the board.
Dashboards tell you something already broke. Observability tells you something is about to break. The difference is the gap between a monitoring system and an immune system.
Most data teams are stuck on monitoring. Here's how to build observability that actually prevents incidents instead of just documenting them.
Monitoring answers: "Is it broken right now?" You define checks. They pass or fail. You get an alert after the damage is done.
Observability answers: "Is it about to break, and why?" You instrument your pipelines so deeply that anomalies surface before they propagate downstream.
| Aspect | Monitoring | Observability |
|---|---|---|
| Timing | Reactive (after the fact) | Proactive (before propagation) |
| Scope | Known failure modes you've coded checks for | Unknown failure modes detected through anomaly patterns |
| Granularity | Table-level (is this table fresh?) | Column-level (has the distribution oforder_amountshifted?) |
| Root cause | "This table is stale" | "The upstream API changed its pagination, causing truncated ingestion at 3am" |
A green dashboard doesn't mean your data is correct. It means none of your predefined checks caught a problem. The failures you haven't anticipated are the ones that reach the board deck.
Every data handoff is a contract. When an upstream system changes a column name, type, or nullability, your pipeline should know before it runs, not after it fails.
What to enforce:
How to enforce it:
Tools: Great Expectations, Soda, dbt schema tests, or custom validation using Pandera or JSON Schema.
Schema checks catch structural changes. They don't catch semantic changes. If order_amount shifts from a median of $45 to $4,500, the schema is fine but the data is wrong.
What to track:
How to track it:
Column-level distribution drift catches the failures that schema checks miss. A table can be structurally perfect and semantically wrong.
Data consumers need guarantees, not best-effort. Define SLOs explicitly:
orders table is updated within 2 hours of midnight UTC, 99% of the time."transactions load contains at least 50,000 rows, and no more than 500,000."customers table have less than 0.5% null rate."Publish these SLOs. Measure against them. Report SLO compliance monthly. When an SLO is breached, treat it like a production incident, not a data team problem.
The most underused pattern in data engineering: stopping a bad pipeline before it poisons downstream tables.
A circuit breaker halts pipeline execution when quality checks fail. The logic is simple:
This requires a cultural shift. Most teams would rather serve stale data than no data. But serving stale data with a freshness indicator is far better than serving wrong data that looks current.
Implementation: dbt's on-run-end hooks, Airflow's ShortCircuitOperator, or custom validation steps that exit with a non-zero code on failure.
It's better to serve yesterday's correct data than today's wrong data. Circuit breakers prevent bad data from reaching consumers.
Data observability shouldn't only run in production. Shift it left:
In development: dbt tests and schema validations run locally before code is committed. Catch issues at the source.
In CI: Pull requests trigger validation against a staging environment with production-like data. New transformations must pass quality gates before merging.
In staging: Full pipeline runs with observability checks. Distribution profiles compared against production baselines. Anomalies flagged before deployment.
In production: Continuous monitoring with circuit breakers. Automated alerting. Historical trend tracking. Incident response runbooks.
| Stage | Checks | Purpose |
|---|---|---|
| Development | Schema tests, unit tests | Catch logic errors early |
| CI | Integration tests against staging data | Validate transformations before merge |
| Staging | Full observability suite against prod-like data | Catch environment-specific issues |
| Production | Continuous monitoring, circuit breakers | Prevent downstream propagation |
| Category | Tools | Notes |
|---|---|---|
| Schema validation | Great Expectations, Soda, dbt tests | dbt tests are lowest friction if you're already using dbt |
| Anomaly detection | Monte Carlo, Bigeye, Metaplane | Managed platforms with ML-based anomaly detection |
| Profiling | Great Expectations, whylogs | Open source column-level profiling |
| Lineage | DataHub, Atlan, dbt lineage | Essential for root cause analysis |
Stop building dashboards that tell you what already broke. Build observability that prevents breakage from reaching consumers. Schema contracts at boundaries. Distribution drift detection at every stage. Freshness SLOs published and enforced. Circuit breakers that stop bad data before it propagates.
The goal isn't perfect data. It's knowing exactly how imperfect your data is, and stopping the worst of it before anyone notices.