Measuring an AI system in business terms
"It feels faster" is not a result. An AI system should be judged the way the business is judged: against a baseline, in numbers people already understand.
Achraf OuahabFounder, ACHRAF OUAHAB MEDIA LLC
3 min read
On this page
In short
- Take a baseline before anything changes.
- Track a small set of business metrics: cycle time, throughput, exception rate, error rate and cost per transaction.
- Agree the targets in the business case, so results are judged in business terms.
Many AI projects report activity instead of results: how many prompts were run, how many documents were summarized, how many people logged in. None of those tell you whether the business runs better.
Start with a baseline
Before anything is built, measure the process as it runs today. You do not need perfect data. A few weeks of honest numbers is enough to compare against. Without a baseline, every improvement is an opinion.
Five metrics that matter
- Cycle time
- How long one piece of work takes from start to finish, including the waiting. This is usually where agentic systems make the biggest visible difference.
- Throughput
- How much work the process completes in a week with the same team. This is the number that shows whether capacity can grow without headcount growing at the same rate.
- Exception rate
- How often work falls outside the rules and needs a person. A falling exception rate means the system handles more of the routine work.
- Error rate
- How often the result is wrong and has to be fixed later. Speed is worthless if errors rise with it.
- Cost per transaction
- What one piece of work costs end to end: people's time, tools and services. This turns the other metrics into money.
Agree the targets up front
The metrics and targets belong in the business case, agreed before work starts. That does two things. It keeps the build focused on what moves the numbers. And it means the result is judged the same way by everyone, from the team doing the work to whoever signs off the budget.
Measure quality, not only speed
Alongside the business metrics, the system's own quality is tested continuously: real scenarios with known right answers, run before launch and after every change. If an update makes the system faster but less accurate, it does not ship.
Keep measuring after launch
Going live is where the work starts. Every action is traced, so the numbers come from what the system actually did, not from a survey. Over time, the trends show where the system can be trusted with more, and which process to extend to next.
If a project cannot tell you its baseline and its five numbers, it is not finished. Those numbers get agreed in the planning step, before anything is built.