Reliable AI needs a useful way to say it does not know

An AI product is not reliable because it answers every request. It is reliable when it can distinguish a supported answer from a plausible guess, show what is missing, and move the user toward a safer next step. That makes “I don’t know” a product capability, not an error message.

Define abstention as an outcome

An abstention policy should start with the cost of being wrong. For each task, decide which of four outcomes the system may produce: answer, answer with a limitation, ask for information, or hand off. The boundary will differ by action. Suggesting search terms can tolerate more uncertainty than approving a refund, changing an account, or advising on health.

This is a coverage decision. The system answers some portion of cases and declines the rest to keep error within an acceptable range. Research on selective prediction formalises that trade-off between coverage and risk. Product teams should measure both. A 98% answer rate is not an achievement if the final 8% of those answers are unsupported.

Write the policy in operational terms:

• Answer when required evidence is present and the request is within the tested task boundary.
• Qualify the answer when evidence supports only part of the request.
• Ask a question when one obtainable fact would materially improve the result.
• Hand off when the action is consequential, evidence conflicts, or the system cannot safely resolve ambiguity.

Test evidence before confidence

A model’s fluent tone is not a confidence measure. Nor is a number the model invents when asked, “How sure are you?” Confidence boundaries should use observable signals from the whole system.

For retrieval-based products, useful signals include whether retrieval returned anything, whether the sources are current and permitted, whether several passages agree, and whether the generated claims can be traced to those passages. For structured workflows, check that required fields pass validation and that tool results confirm the requested action. For classifiers, calibrate scores against held-out examples from the intended environment.

The interface should expose the reason for abstention without dumping internal mechanics on the user. “I couldn’t find a cancellation rule for annual plans in the approved help centre” is useful. “Low confidence” is not. It names the missing evidence and tells the user what kind of correction could change the outcome.

This matters because confident falsehoods are a known property of generative systems, not an unusual edge case. The NIST Generative AI Profile identifies confident false content, automation bias, and weak information integrity as connected risks. A polished answer can increase harm when the evidence beneath it is thin.

Ask only questions that change the route

A follow-up question is valuable when its answer changes what the product will do. Before asking, the system should know which decision depends on the missing value.

Good questions are narrow: “Which invoice do you mean?” or “Should I use the policy effective on the purchase date or today?” They offer choices when the valid set is known and explain why the detail matters. Avoid open-ended prompts such as “Can you provide more context?” They transfer the system’s diagnosis work to the user.

Set a question budget. After one or two failed clarification attempts, repeating the loop usually adds friction rather than information. The next state should be a constrained alternative, such as showing relevant documents, saving a draft, or offering human review.

Products also need to distinguish unavailable evidence from conflicting evidence. No source found calls for a request or a decline. Two authoritative sources that disagree call for a visible conflict and often a person who can decide which rule governs.

Make handoff preserve the work

“Contact support” is not a handoff. It is an exit.

A useful handoff carries the user’s goal, supplied facts, retrieved evidence, attempted actions, uncertainty reason, and any consent needed to share that context. The person receiving it should see a concise case summary and the original material, not an opaque model interpretation. The user should know whether the case was submitted, who owns it, and what happens next.

Routing should follow the reason, not merely the topic. Missing permission may go to an account owner; conflicting policy to operations; a safety concern to a trained specialist. If no person is currently available, preserve the case and give the user a stable reference rather than pretending the conversation remains live.

Treat handoff quality as part of model quality. Track whether reviewers received enough context, whether they had to repeat questions, and whether the abstention reason matched the eventual resolution.

Put the boundary into evaluation

The test set must include cases the system should refuse. Build it from answerable requests, requests with missing evidence, contradictory sources, out-of-scope questions, malformed inputs, stale documents, prohibited actions, and cases requiring human judgment.

Score more than answer accuracy. Measure unsupported-answer rate, correct abstention rate, unnecessary abstention rate, question usefulness, evidence attribution, handoff completeness, and performance at each coverage level. Review these by task and risk class; one aggregate score can hide failure in the cases that matter most.

Thresholds should be selected against those evaluations, then monitored after changes to models, prompts, retrieval, tools, or source content. A threshold copied from a demo has no meaning until it is calibrated on representative cases. Log the signals and route chosen so reviewers can explain why the system answered or stopped.

This is how Alfcode describes its AI product development practice: retrieval over approved data, evaluation against real cases, guarded tool use, and a clean path to a person. The same separation appears in Autopilot, where an agent’s software judgment is not the final control; on-chain rules reject actions outside defined constraints. The practical lesson extends beyond financial agents: prompts may guide behaviour, but enforceable boundaries must sit in the surrounding product.

Before shipping, run one decision test: remove a required source, introduce a conflicting source, and request a consequential action. The product should name what prevents an answer, ask only for information that can resolve it, and preserve the full case when a person must take over.