AngoraTek FR
← Back to Blog
Data Engineering 7 min read

Your Dashboard Is Green and the Numbers Are Still Wrong

Monday, in the meeting where the number matters: slide three says 41,280 permits issued this quarter. A refresh at 6:00 a.m. and another at 6:07 a.m. both show green arrows on the DAG, and the two disagree by 26,000. Nothing failed. The orchestrator is telling the truth, which is the whole problem.

The status a scheduler reports answers exactly one question: did the code run. It does not answer the question anyone upstairs is asking, which is whether the table now contains the truth. Closing that gap is a design task, not a personality trait of the data team, and most of it is a small set of checks that run after the job and treat the output as evidence rather than as a promise. This post walks the four we never leave out, on one worked example, a permits pipeline fed by a vendor export.

The DAG testifies to execution, and to nothing else

Think about what a green checkmark actually observed. The process started, retried nothing, and exited with code zero. That is the entire testimony, and it holds equally for a table full of the right rows and for a table empty because a filter got one value too tight two deploys ago. The orchestrator is not lying to you. It is answering a different question than the one you asked, and its answer is genuinely true.

The emptiest success we see most often is the over-tight filter. A join key gains a normalization (lowercase, maybe), the vendor keeps sending mixed case, the join drops every row, and the pipeline reports success on zero rows while downstream queries show a healthy yesterday and a dead today. A DAG cannot see that, because from its seat it did its job. A pipeline whose every failure mode is visible to the scheduler is a pipeline whose failures all crash the process, and the ones that do not crash, the wrong answer, are the ones with business consequences.

Row counts pass too easily to be the whole test

The standard countermeasure, reconciling row counts between source and sink, is the first thing we reach for and the first thing that betrays. Zero rows reconcile perfectly against zero rows. Duplicates reconcile: a bug that duplicates rows upstream and an accidental dedupe downstream agree on the total while every individual row is double. And a pipeline can move exactly the right number of rows with every address field null, because a column silently stopped populating and nothing in a count looks at contents.

What we reconcile instead are invariants that a bug struggles to satisfy by accident. Distinct primary keys against row count, which catches duplication. The sum of a measure with known behavior, say total permit fees, compared across the pipeline boundary, which catches a value that shifted or a filter that ate revenue. Min and max event dates inside the expected window, which catches the stale partition quietly served as if current. None of these checks knows what the data should be; each one makes an error expensive to produce, which is a different and better bargain than asking a count to certify contents.

The invariants run after the job, against the sink, and they can fail the pipeline: a failing invariant blocks the publish step, which is the only way a wrong table stops being the table everyone reads.

Null rates are the cheapest sensor you already have

One column, one line of SQL, and it has caught more vendor-side changes than every schema diff we have ever filed. A null-rate sensor records, per table and per column, the share of null values at each load. A column that has run 0.2 percent null for weeks and arrives at 31 percent null has not failed anything; the export changed, quietly, and now every geocode lookup downstream is missing a third of its inputs.

SELECT load_date,
       round(100.0 * count(*) FILTER (WHERE address IS NULL)
             / count(*), 2) AS address_null_pct
FROM raw_permits
GROUP BY load_date
ORDER BY load_date DESC;

The alert condition is drift against its own history, not a fixed threshold guessed in a meeting. A column that has always been 40 percent null is fine. The same column that moved from 4 to 40 in one load is an incident. We store one row per table per load, and the jump between the last two rows is what triggers the page. This is the same validation-loop posture we argue for in geocoding a public census, and for the same reason: cheap checks run continuously beat expensive checks run at the end of a quarter.

The same harness extends naturally: distinct counts of keys that should be near-unique, and row counts by partition in place of table-wide totals. One sensor per family, tuned to the column where a silent change would cost the most, is worth any monitoring product we have demoed with a straight face.

Freshness is an SLO, so give it an owner and a page

A timestamp on a dashboard tells the reader when the table updated. It does not tell them whether that update carried current data, and it assigns nobody the duty of noticing. A freshness SLO does both: per table, freshness is the age of the newest event data in the sink, now minus max(event_time). The SLO states how old the table is allowed to be, and a breach pages a named person, with a runbook step that says what to check first.

Setting the number is a one-time conversation, and it is where the vendor's publish schedule and the business need meet. The permits export lands nightly at 5:00 a.m., so the sink should be hours old at the morning meeting, not days. Say it as a promise with a name on it, per the SRE practice Google documents on implementing SLOs: the target, the measurement window, and the consequence when it misses. A freshness number without a consequence is a wish with a widget attached.

Then let the SLO fail loudly. A breach at 6:00 a.m. pages, rather than letting the number sit stale until the next meeting discovers it. The failure mode this catches is precisely the silent kind, a pipeline that ran and produced something: green arrows over correct code, serving yesterday's data.

What reconciliation cannot catch

The invariants share one blind spot, and it deserves its own sentence: they compare the pipeline against itself. A vendor that begins sending a wrong but internally consistent field, permit fees shifted into estimates rather than actuals, reconciles cleanly at every stage we monitor, because the error is upstream of every check. Only a comparison with a definition, an expected total the finance team holds independently, catches that class of error. We keep one hand-run spot-check on the small tables. The cost is minutes a month, and it is the only check in the whole apparatus that knows what a permit fee should be.

The sensors also have their own failure modes, which are quieter than the faults they watch. A renamed column turns a null-rate sensor into a device reporting 100 percent forever, and without an alert on the sensor itself, the monitor becomes the newest source of false confidence. Alert on the monitor: a sensor that stops producing comparable history is a breach of its own.

And every check we list is only as alive as its owner. Orphaned monitors drift. Thresholds from two years ago keep firing at irrelevance until someone silences the channel, and a page that fires twice for nothing trains the team to sleep through the third. The census of monitors is as real a chore as the census of columns, and it is the price of the whole design.

Green is a statement about the code. The numbers need their own witnesses.

AngoraTek designs these checks into pipelines we hand over, not as bolt-ons. If your team is currently the only sensor you have, bring us the pipeline and we will hang the witnesses before the next Monday meeting.

Thanks for reading this far. If this sounds like the problem you're wrestling with, tell us about it. The team reads every message.

Start a conversation