Scoring against your prediction
Predicted against actual, over the window you chose before you started.
On day 14, before anyone changed how they worked, you wrote down what would happen at the bottleneck. The level now, the level you expected, the window you'd measure it over, how far things might dip first, and the two lines that would count as success and as failure. Then you locked it. Today you find out. This lesson is about scoring that prediction honestly, including when the answer isn't the one you hoped for.
Most AI work is never scored
Most organisations can't say whether their AI work succeeded, because they never said in advance what success would look like.
of AI initiatives delivered the return that was expected of them.
IBM IBV, 2025
of CIOs have formal measures of success for their AI work.
Foundry, 2026
of CIOs say their AI initiatives met or beat their business goals.
Foundry, 2026
With no prediction, almost any result can be read as progress. Someone finds a number that went up, and the work is declared a success. That's how a company ends up with a dozen successful pilots and nothing to show on the profit line.
The organisations that do get value tend to fund an outcome with dates attached, and check it on those dates.
“We don't fund ‘build a model,’ we fund ‘reduce returns by 8% on this category’ with checkpoints at 90, 180, and 270 days.”
Why it had to be written down first
A prediction written after the result always comes true. The target slides to wherever the line ended up, and the result is called a success. That's why the prediction was signed and locked on day 14, before Improve started. A locked prediction can prove you wrong, and that's exactly what makes it worth something when it proves you right.
Things do change after day 14. A system is replaced, a measure turns out to be wrong, the window has to move. None of that quietly replaces the original. Each change goes in the readout as a change, with the reason, and the original prediction still gets scored.
Read the window, not the week
Work usually gets worse before it gets better. People are learning new templates, a step is being redesigned, someone is in training during hours they'd normally spend on the work. That's why your prediction included an expected dip: how far, and for how long.
The rule is simple. Keep the dip you wrote down, and don't read it as failure until the window closes. A leadership team that panics in week 3 kills the change before it has had a chance to work. A team that shrugs off a dip twice as deep as predicted misses a real problem. The prediction tells you which one you're looking at.
Score it, then take Improve's exit
Use the same system you used on day 0, and the same measure. If you took readings every two weeks, add them: they draw the real path, including the dip. Then enter the level over the window, and score it against what you locked.
Scoring ends in one of three exits. They're the three the framework gives Improve.
- It moved. Go back to Find. Fixing this bottleneck has exposed the next one, and lesson 4 names it.
- It didn't move, and the gap is worth paying for. Raise it at the front door for Build, with a one-page case. Improving without spending has done what it can.
- It didn't move, and it isn't worth paying for. Keep improving, or accept the level you've reached and name the next bottleneck.
“Worth paying for” is a judgement, but it isn't a guess. In module 4 you put a value on one more unit through the bottleneck. Compare what the gap is worth with the full running cost of closing it, not just the price of building something.
What counts against the framework
The pilot kit sets these results before a pilot starts. If any of them happens, the readout says so in its first line, as plainly as anything that supports the framework. Your own failure threshold joins the list.
- The bottleneck moved as predicted, but the aim's outcome didn't. Either the core claim is wrong here, or the wrong bottleneck was picked. Either way, Find failed.
- Disclosure covers less than half of the AI tools that monitoring found. The promises didn't make telling the truth safe.
- Trust in the disclosure promise fell between day 0 and day 90.
- Most freed hours were never converted. The rule on where freed time goes didn't hold. Lesson 2 checks this.
- The front door took a median of more than 10 working days to give a first answer.
- A shared or high-stakes item ran with no owner.
- Rework rose at the bottleneck or the step after it. AI output flooded the bottleneck.
You'll bring all of these together in your readout in lesson 5.
A result is inconclusive when a measure couldn't be collected, the window was cut short, or the organisation changed in the middle of the loop. Record it as inconclusive, and say why. Never round it up to a success, and never fill the gap by swapping in a measure that happens to look better.
The window has closed. Output at the bottleneck averaged 24 a week, against a prediction of 35 and a failure threshold of 25. The person who leads the bottleneck points out that cycle time improved a lot, and suggests the readout leads with that instead.
What do you do?
Score what you wrote down on day 14, not what you wish you'd written.
You predicted 35 quotes a week over days 60 to 90, with a dip of up to 15% in weeks 2 to 4. In week 3, output is down 12%, and a director wants to stop the change. What do you say?
Mark it complete. Your answers are already saved in this browser.