AI Confidence Scores: How to Interpret Them

A Number That Has to Earn Its Place

Confidence scores are easy to produce and easy to misuse. A tool that labels a conversion “92% confident” has told you almost nothing unless you know what the number is measuring, what it was calibrated against, and what you are supposed to do differently at 92 versus 80.

Here is how to read ours — and, just as usefully, how to interrogate anyone else’s.

What the Score Measures

Ours is not a probability that the output is correct. No tool can honestly claim that, because correctness depends on intent the tool cannot see. It is a measure of how completely the construct was recognised: did every element of this object map onto a known, tested rule, or did some part of it fall back to a best effort?

• High. Every construct matched a deterministic rule with a test behind it. A VARCHAR2(50) column, a straightforward B-tree index, an NVL call. Review these in bulk, by sampling.
• Medium. Converted, but with a semantic difference worth knowing about — a %ROWTYPE parameter becoming RECORD, or a CHAR column where comparison semantics shift. Read the warning; usually accept.
• Low. A rule applied but the construct has no true equivalent — BITMAP falling back to B-tree, FAST refresh becoming scheduled refresh. These need a decision, not a review.
• Refused. Not a low score at all. No output, an explicit gap entry, and a named reason. This is the most valuable state in the system.

Why “Refused” Beats a Low Score

The temptation with any scoring system is to always emit something and let the number carry the caveat. It doesn’t work: people read output and skim scores. Plausible-looking SQL with a 40% badge next to it will be copied into a target schema by someone in a hurry, and the badge will not save them.

So below a threshold we stop producing converted output and produce a gap entry instead. It is a worse-looking number on a slide and a much better outcome in a migration.

The Question to Ask Any Vendor

What was this calibrated against? A confidence score is only meaningful if someone checked, on a known corpus, that the high-confidence items were actually right. Ours is anchored to a benchmark dump of roughly 4,000 objects with known expected outcomes, run on every change, where a regression fails the build.

If the answer is a shrug, the number is decoration.

Using Scores to Plan Review

High      3,742 objects   sample ~5%, spot-check the pattern
Medium      168 objects   read the warning, accept or adjust
Low          62 objects   decide per object, senior engineer
Refused      28 objects   design work: no equivalent exists

The value of the score is not the number — it is that it turns “review four thousand objects” into ninety that need real attention and a sampling strategy for the rest. Reviewing everything equally is how review budgets get spent on the easy 95%.