Predicted vs Actual: How We Check Whether Our Intent Engine Is Asking Useful Questions
The Intent Engine can forecast a question’s value—and production evidence can show whether that forecast was too high, too low or well calibrated.
Quick answer
A question can look valuable in theory and contribute less in real journeys—or the opposite. Comparing predicted information gain with measured information gain helps reveal that bias before changing question ranking.
Why this matters
This guide is part of our Adaptive Intent & Verified Evidence series. It explains a production design principle behind a website that tries to understand the customer with fewer questions while keeping safety, external evidence and real requests governed by deterministic rules and customer or human confirmation.
Prediction and selection are not the same
The overall selector score can include safety importance, required-slot value and friction. Predicted information gain is narrower: it forecasts how much decision and booking information the answer is expected to add. Keeping these separate prevents a mandatory safety question from being incorrectly judged as a bad prediction.
Measure the realised effect
After the customer answers, the system calculates a bounded measured value from confidence change, recommendation change and movement toward the Ready state. The before/after record is linked by the governed question ID.
Overestimated, underestimated or well calibrated
If measured gain is consistently below predicted gain, the engine may be overestimating that question. If it is consistently higher, the engine may be underestimating it. A small difference can be treated as well calibrated.
Calibration needs a sample threshold
A single mismatch proves very little. The system should wait for enough paired predicted-and-measured examples before presenting a bounded calibration proposal to an administrator.
No automatic production mutation
Calibration evidence is not permission to self-edit. An authenticated human reviewer decides whether to approve, reject or collect more evidence. Approved changes remain bounded and versioned so they can be audited and rolled back.
Better calibration means fewer wasted questions
The benefit is not a cleverer dashboard. It is a more accurate estimate of which question is worth asking next—so customers are less likely to see questions that add little value.
Frequently asked questions
What does a negative calibration bias mean?
It means measured information gain was lower than predicted, suggesting the question was overestimated in that sample.
Does calibration change safety priority?
No. Safety priority remains policy-controlled even when the system reports calibration statistics.
Why not use the full selector score for calibration?
Because selector score contains factors such as safety and friction that are not the same as a forecast of information gain.
Can calibration be rolled back?
Yes. Approved calibration should be stored as a versioned override rather than rewriting the base question definition.
How this fits with Intent Intelligence and QuoteMe
The Intent Engine is the decision controller: it interprets governed facts, decides whether another clarification materially helps and returns a readiness state. The QuoteMe Journey Controller carries approved context across the website. QuoteMe remains the only component that creates a real request after the customer reviews and submits it.
Could your website ask less and understand more?
Tell us where customers repeat themselves, choose the wrong service or need clearer evidence before the next step. We can review whether intent-driven logic, automation, structured evidence or something simpler fits the problem.