How Do You Know Your Data Quality Agent Is Right?

Agent verification is the practice of confirming that an AI system's finding traces back to records you can open yourself, instead of accepting prose that merely sounds correct. Every data quality vendor now ships an agent. Almost none of them tell you how to check its work.

That gap is the whole problem. An agent that says "fct_revenue looks stale because an upstream sync failed" is either doing real correlation or generating a sentence with the right shape. From the outside, those two things are identical. They read the same, they arrive in the same Slack channel, and one of them is worthless.

I build one of these agents, so treat what follows as interested but specific. Everything below describes machinery I can point at.

Why is "the agent said so" not an answer?

The failure mode is not that the model lies. It's that the model is fluent about things it half-knows, and fluency is the exact signal humans use to judge competence.

Here's the concrete version. You ask why a table went stale. Behind the scenes the system pulls rows from a few tables: schema change events, alert history, freshness records, metric snapshots, lineage edges. Then it does one of two things. It stuffs those raw rows into a context window and asks a model to write an explanation, or it composes an answer from a structure it built before the model was ever called.

The first approach produces an answer that name-drops events without linking to them. The model saw twelve rows, mentioned four, invented a causal story connecting them, and dropped the eight that didn't fit the narrative. You can't tell which four were real. Neither can the person who forwards the explanation to a stakeholder.

Data engineers already ran this experiment with dbt tests. A test that fires with severity warn produces a line in a log that nobody reads, and everyone treats a green build as a passing build. The lesson was that an unverifiable signal decays into no signal. Agents are the same shape of problem at higher fluency.

So the question worth asking a vendor isn't "how accurate is your agent." Accuracy claims are unfalsifiable without your data. The question is: what can I click on?

What does a verifiable finding actually look like?

The structure we settled on is a capsule: a typed object with a fixed shape that every investigation returns, where each piece of evidence carries the identifier of the row it came from.

{
  "investigation_id": "5f2c1a7e-...",
  "question": "Why is fct_revenue stale?",
  "root_cause_hypothesis": "Schema change on upstream table stg_orders: column_type_changed",
  "confidence": 0.75,
  "triggers": [
    {
      "event_type": "schema_change",
      "confidence": 0.75,
      "timestamp": "2026-08-27T19:04:00Z",
      "summary": "Schema change on upstream table stg_orders: column_type_changed",
      "source": "schema_change",
      "source_id": "9b41d0c2-...",
      "details": { "column_name": "order_total", "is_breaking": true }
    },
    {
      "event_type": "freshness_cascade",
      "confidence": 0.85,
      "summary": "Upstream table raw_orders is also stale (11h since last update)",
      "source": "lineage",
      "source_id": "c7a9e310-..."
    }
  ],
  "consequences": [ "..." ],
  "suggested_fix": "Review the type change on order_total before rerunning the model",
  "open_questions": []
}

The important field is source_id. It's a public UUID for a row that exists in the database, and it resolves through one uniform lookup no matter which of the five sources it came from: intelligence records, schema change events, alerts, metrics, or lineage nodes. In the web agent, each evidence row renders as a link. You click the schema change and land on the schema changes page for that asset, looking at the event itself.

That's the entire trick, and it's less clever than it sounds. The capsule is a typed projection of a row the investigation service already wrote. It doesn't re-derive anything. It doesn't ask a model to summarize. It takes structured output that already existed and gives it a shape you can navigate. We wrote about the mechanics of this in citations for your data incidents.

The consequence matters more than the mechanism. If a claim has no source_id, it didn't come from a row. There's no ambiguity to argue about. Either the evidence points at something, or the answer is prose.

How is that confidence number produced?

Here is where most vendor documentation goes vague, so let me be specific about ours.

Confidence on a piece of evidence is a fixed prior attached to the type of correlation that found it. Not a model's self-assessment, not a learned score. A constant in the code, chosen by a human, per correlation type.

The overall confidence on the capsule is the confidence of the single strongest event. Nothing more elaborate. The events sort by confidence descending, the top one becomes the root cause hypothesis, and its confidence becomes the capsule's confidence.

We considered having a second model grade the first model's answer, and rejected it. LLM self-grading is documented as overconfident, it costs a call per investigation, and a number that goes up when the model feels good about itself isn't a measurement. Rules are deterministic, free, and I can explain them to a customer in a table, which is what I just did.

What is the confidence number actually measuring?

Read that table again and you'll notice something most vendors would rather you didn't.

Because the capsule's confidence is the top event's fixed prior, a capsule whose strongest signal is a same-table schema change always reads 90%. A capsule whose strongest signal is a metric anomaly always reads 60%. The number doesn't move with how much evidence piled up, and it doesn't move with how likely the hypothesis is to be correct.

It's a category label wearing a percentage sign. It tells you what kind of evidence was found, not the probability that the answer is right.

That's a real limitation and I'd rather say it than have you discover it. The number is still useful, because "this is a direct schema change on your table" and "this is a metric that wobbled in the same 24 hours" are genuinely different qualities of evidence and you should treat them differently. But if you read 90% as "nine times out of ten this is the cause," you're reading something that isn't there.

The UI bands it deliberately, which is an admission of the same thing:

Three buckets is about the resolution the underlying signal supports. Any tool showing you 87.3% confidence on a root cause is showing you false precision, and you should ask what produced the digits.

Can you see which detector made the call?

Same principle one layer down. When an anomaly detector says a value is out of range, you should be able to see which detector, what range, and what it expected.

Our metric detection returns the verdict alongside the machinery: whether it used Prophet-based forecasting or fell back to a simple standard deviation threshold, the expected value, the expected range, the anomaly score, and which seasonality patterns it detected. The fallback triggers below 14 historical data points, because forecasting a seasonal model on 9 observations produces confident nonsense.

Sensitivity is a confidence interval, and it maps to a z-score by the normal quantile function rather than by a lookup table someone tuned:

None of that is proprietary and none of it should be. If a tool won't tell you whether a finding came from a forecast model or a two-line standard deviation check, you can't calibrate how much to trust it, and you'll end up trusting all of it equally. That's how alert fatigue starts. The general shape of these methods is covered in the anomaly detection guide.

The honest framing: a threshold fallback on 11 data points is a rough check, and knowing it's rough is what lets you ignore it correctly.

What happens when the agent is wrong?

It will be wrong. Black Friday, a backfill, a marketing campaign, a schema migration you scheduled yourself. The volume triples and the detector does exactly what it should do, which is flag it, which is exactly what you don't want on the fourth identical alert.

The test isn't whether the agent is wrong. It's whether your disagreement survives.

The mechanism we ship is snapshot exclusion. Mark a data point as expected and it's excluded from the baseline, with the reason, the user, and the timestamp stored alongside it. The next detection run filters those points out before the model sees them.

# Mark a known spike as expected so it stops poisoning the baseline
curl -X PATCH \
  "$ARMOR_API/api/v1/assets/$ASSET_ID/metrics/$METRIC_UUID/snapshots/$SNAPSHOT_ID/exclude?excluded=true&reason=Black+Friday+traffic" \
  -H "Authorization: Bearer $ARMOR_API_KEY"

Two properties make this a real feedback loop rather than a mute button. The exclusion is reversible, so passing excluded=false puts the point back. And the excluded points stay visible on the metric history chart, marked as excluded, so a year from now somebody can see that the November spike was real and deliberately set aside.

Compare that to the common alternative, which is snoozing an alert. Snoozing hides the symptom and leaves the bad point in the baseline, where it widens the expected range and quietly makes the detector worse at its job forever. That's the difference between correcting a system and silencing it.

The question to ask a vendor: when I tell your agent it's wrong, does anything change in how it computes the next answer, or did I just close a notification?

The four-question audit

Here's the framework, and it's short because it needs to survive a vendor call.

1. Does every claim cite a record I can open? Not a link to a dashboard. A link to the specific row that produced the claim. Ask for a finding with a citation that 404s and see what the UI does.
2. Do I know which method produced the number? Forecast or threshold, which window, how much history. A tool that hides this is asking for uncalibrated trust.
3. Do I know what the confidence number measures? Ask what the number would be for two different findings that are both certainly true. If the answer is "different numbers," it's measuring evidence type, not probability. That's fine, as long as they say so.
4. Does my disagreement change the next answer? Mark something as expected, then check whether the baseline moved.

Score each one. Two points if the tool does it and documents it, one point if it does it and you had to ask, zero if the answer is a deflection.

The middle column is where most of the market sits right now, and it's the dangerous one. Dashboard-era tools don't make claims, so they can't hallucinate. A model summarizing your warehouse can, and it does it in complete sentences. We drew the same distinction from a different angle in what separates a real agent from a wrapper and compared specific vendor approaches in data quality agents compared.

Notice that three of the four questions have nothing to do with model quality. A better model doesn't fix an unciteable answer. The verifiability is an architectural property, decided before the model is called, and no amount of prompt engineering adds it later.

How do you run the check yourself?

Don't take a vendor demo's word for it. The demo is on their data, where they know the answers.

The version worth doing is a replay on an incident you already understand. Pick a failure from last quarter where you know the root cause cold. Point the agent at the affected table and ask why it broke. Then grade the answer against what actually happened.

from anomalyarmor import Client

client = Client(api_key="...")

capsule = client.investigations.explain(
    asset_id="fct_revenue",
    question="Why did this table go stale on August 27?",
)

print(capsule.root_cause_hypothesis, capsule.confidence)

for evidence in capsule.triggers:
    # Every row should name a source and an id you can open
    print(f"{evidence.confidence:.2f}  {evidence.source}  {evidence.source_id}")
    print(f"      {evidence.summary}")

Three outcomes and all of them are informative. The agent names the cause you know is right, and every trigger resolves to a real row: that's the good case, and now you know what its evidence looks like when it's correct. The agent names something plausible but wrong, and the triggers still resolve: the correlation was real, the ranking was off, and you've learned that the top hypothesis deserves scrutiny. The agent writes something confident with no citations underneath: you have your answer about the tool.

Run it on five incidents. You'll learn more than any benchmark table will tell you, because the failure modes are specific to your warehouse's shape. A warehouse with 20 tables and no lineage produces thin capsules regardless of how good the agent is, and that's a fact about your setup rather than the software.

Budget an hour for this in any evaluation. If your loaded engineering cost is $80 to $150 an hour, an hour of replay against known incidents is the cheapest procurement diligence available, and it's more predictive than a 30-day trial where nothing breaks.

What this still doesn't prove

Four things this approach doesn't give you, stated plainly, because a post about verification that oversells its own verification would be funny in the wrong way.

A citation proves provenance, not causation. Every trigger in a capsule can resolve to a real row and the hypothesis can still be wrong. The schema change was real, it happened 40 minutes before the failure, and it had nothing to do with it. Citations rule out fabrication. They don't rule out coincidence, and time-correlated evidence is exactly where coincidence lives.

Capsules are computed on demand, not stored. We deliberately don't persist them, which keeps a migration, a retention policy, and a privacy review off the table. The cost is that you can't diff how an explanation changed between Tuesday and Thursday, and you can't build regression tests on answer quality over time. That's a real feature we don't have.

Older investigations degrade. Evidence rows written before the citation fields existed project into capsules with an empty source, so they render as summaries without links. The shape is backward compatible on purpose, which means the guarantee is "new findings cite" rather than "all findings cite." Check the date on anything you're auditing.

A resolvable identifier is a contract, not a guarantee. The registry that maps each evidence source to its lookup exists so citations resolve uniformly. It isn't a substitute for you clicking one. If a cited row was deleted after the investigation ran, the link 404s, and the UI says so rather than pretending otherwise.

The reason to be this specific about limits is that the alternative is the vendor position, which is that the agent is accurate because the vendor says so. That claim is unfalsifiable and everybody knows it, which is why nobody believes it and why the honest version is a competitive advantage.

The agents are going to get better. The verification surface is what determines whether you can tell.

Frequently asked questions

How do I verify an AI data quality finding?

Click through to the underlying record. A verifiable finding carries an identifier for the row that produced each claim, so you can open the schema change event, alert, or metric snapshot and confirm it exists and says what the agent said it says.

What is an evidence capsule?

A typed object returned by an investigation containing the question, a root cause hypothesis, triggering evidence, downstream consequences, a confidence value, and a suggested fix. Each piece of evidence names its source and the UUID of the row it came from.

Should I trust an LLM's confidence score?

No, if the model generated it by grading itself. Self-graded confidence is well documented as overconfident and it moves with the model's fluency rather than with the evidence. Ask how the number is computed before you use it to prioritize anything.

What does a 90% confidence root cause actually mean?

In our implementation, that the strongest correlated event was a schema change on the table itself within the last 48 hours. It's a fixed prior for that category of evidence, not a probability that the hypothesis is correct.

Why use fixed confidence priors instead of a learned score?

They're deterministic, cost nothing to compute, and can be published in a table a customer can argue with. A learned score would need labeled incident outcomes we don't have at volume, and an unexplainable number is worse than a coarse one.

How does an agent avoid hallucinating a data incident?

By composing the answer from structured rows before the model is involved. The correlators query real tables, the capsule is a projection of what they found, and the model's job is limited to writing prose over a structure it can't add rows to.

Can I tell which anomaly detection method fired?

You should be able to. Ours returns the detection method alongside the verdict, either a Prophet-based forecast or a standard deviation fallback, plus the expected value and range. The fallback runs below 14 historical data points.

What happens if I mark an anomaly as expected?

The snapshot is excluded from the baseline with the reason and user recorded, and the next detection run filters it out before computing expectations. It's reversible, and excluded points stay visible on the history chart.

Is excluding a data point the same as snoozing an alert?

No. Snoozing hides the notification and leaves the bad point in the baseline, where it widens the expected range permanently. Excluding removes it from the calculation, which is the correction you actually wanted.

How do I evaluate a data quality agent before buying?

Replay incidents you already understand. Pick five past failures with known causes, ask the agent why each table broke, and grade both the hypothesis and whether every cited record resolves. An hour of this beats a quiet 30-day trial.

Does more context in the prompt make the agent more reliable?

Not in a way you can verify. A larger context window means the model saw more rows, not that it cited the right ones. The reliability comes from the structure you hand it, not the volume.

What can citations not tell me?

Whether the correlation is causal. Every cited row can be real and the hypothesis can still be coincidence, which is most likely when the strongest evidence is something that merely happened nearby in time.

Do investigations get stored so I can compare them over time?

Not in our current implementation. Capsules are computed on demand from rows that persist in their own tables, so there's no history of how an explanation changed. Persisting them is a known follow-up, not a shipped feature.