Measuring What Matters

Measuring What Matters

Organisations measure an extraordinary number of things: sales volumes, response times, customer satisfaction, processing costs, staff turnover, error rates, website visits, complaints, project delivery, system availability and dozens — sometimes hundreds — of other measures that find their way onto reports and management dashboards. But the existence of a measure does not make it important.

One of the most useful contributions a Business Analyst can make is to help an organisation distinguish between what can be measured and what actually matters. That requires us to work backwards. Before asking what should appear on a dashboard, we need to understand what the organisation, department, service or initiative is trying to achieve. Only then can we identify the things that must go right, decide what evidence would tell us whether they are going right, and establish what level of performance is acceptable.

Start With Purpose

Suppose you are asked to help a department define its Key Performance Indicators. The temptation is to begin by asking, "What should we measure?" That question comes too early. A better starting point is "Why does this department exist?"

That may sound obvious, but it is surprising how difficult the question can be to answer. You may hear, "We support the business." But support it how? What does the department provide? Who depends upon it? What would stop happening — or start going wrong — if the department disappeared tomorrow?

A useful purpose statement should express, in straightforward business language, what the department exists to contribute. For example:

The Customer Service Department exists to resolve customer problems so that customers can continue to use our services successfully.

That statement may need refinement, but it gives us something enormously important: a reason for the department's existence. We can now begin asking what must be true if it is performing that role successfully.

Critical Success Factors: What Must Go Right?

Once the purpose is understood, we can identify the Critical Success Factors, usually abbreviated to CSFs. A Critical Success Factor is something that must go right if the organisation, department or initiative is to succeed. It is important, however, to distinguish a CSF from something the department merely does.

Consider a Customer Service Department. Answering telephone calls, responding to emails, recording cases, producing reports and holding team meetings are all activities. The department may need to perform every one of them, but performing them does not necessarily mean it is succeeding. A team could answer every telephone call and still give customers useless answers. It could process thousands of cases while repeatedly failing to resolve the underlying problems. It could produce immaculate monthly reports about a service that customers hate.

Critical Success Factors should therefore concentrate on outcomes rather than activities. For our Customer Service Department, possible CSFs might include:

• customer problems are resolved effectively;
• customers receive a timely response;
• cases are handled accurately;
• repeat contact is minimised; and
• sufficient capable staff are available to meet demand.

These statements are much closer to genuine definitions of success. They describe conditions that need to exist rather than simply work that people perform.

Finding the Critical Factors

CSFs should not be invented by a Business Analyst sitting alone with a spreadsheet. They emerge from investigation and discussion with the people who understand the organisation, its customers, its risks and its objectives. Useful questions include:

• What must consistently go right for this area to succeed?
• What would cause serious problems if it went wrong?
• What would senior management notice immediately if it failed?
• What do customers depend upon us to provide?
• What causes the greatest operational or financial risk?
• What would make us say that this department was performing exceptionally well?

The word critical matters. If everything is declared critical, nothing is. A department might perform hundreds of activities and monitor dozens of operational measures, but its genuine Critical Success Factors should normally be a much smaller set. Four to eight is often enough to force useful prioritisation without pretending that every important consideration can be reduced to a single factor.

From CSF to KPI

A Critical Success Factor tells us what matters, but it does not yet tell us whether we are achieving it. For that we need evidence, and this is where Key Performance Indicators — KPIs — enter the picture. A KPI is a measure that provides evidence about performance against something important.

Suppose one of our CSFs is customer problems are resolved effectively. How could we tell whether that is happening? Possible indicators might include the percentage of cases resolved on first contact, the percentage reopened after being marked resolved, customer satisfaction following case closure, the number of repeat contacts concerning the same problem, or the percentage of cases resolved within an agreed service period.

There may therefore be more than one KPI associated with a CSF. That is often desirable because business performance is rarely captured adequately by a single number. The important point is that each KPI should provide meaningful evidence about the success factor rather than simply measuring an activity that happens to be easy to count.

Beware the Convenient Measure

Organisations have a natural tendency to measure what is easy to count, but that is not the same as measuring what matters. Suppose a call centre can easily measure the number of calls handled by each member of staff and therefore sets a target encouraging employees to handle more calls. Performance apparently improves as call volumes per employee rise.

But what happens if employees achieve the target by ending calls quickly without properly resolving customer problems? The measure has improved while the service has deteriorated. This is a fundamental danger of performance measurement: once people know that a number is being used to judge performance, their behaviour may change in ways that improve the number without improving the underlying outcome.

Every KPI Needs a Direction

One of the simplest questions in performance analysis is also one of the most frequently overlooked: "Which way is better?" For some measures, higher is better; for others, lower is better. Customer satisfaction and system availability would normally be expected to increase, while complaint rates, transaction errors, costs and regulatory breaches would normally be expected to decrease.

Other measures require more care because neither continual increase nor continual decrease represents success. Inventory, for example, may need to remain within a range: too little causes shortages, while too much ties up cash and storage space. Staff utilisation provides another example. Very low utilisation may indicate wasted capacity, but attempting to drive utilisation permanently towards 100% may leave no capacity for unexpected work, training, improvement or absence.

For some indicators, therefore, on target, within tolerance or stable may be more meaningful than simply higher or lower. The direction of improvement should be stated explicitly; otherwise a report can tell us that a number changed without telling us whether the organisation should be pleased or concerned.

Define the KPI Properly

A label is not a definition. Imagine a management report containing a KPI called Customer Complaints. Does that mean the number of complaints received during the month, the number still open at month end, complaints per thousand customers, or complaints per thousand transactions? Are only formal complaints included? If one customer makes three complaints, does that count once or three times?

Without a precise definition, different people can calculate what appears to be the same KPI and obtain different answers. A useful KPI definition should therefore identify the KPI name, the CSF to which it relates, its precise definition or formula, its unit of measurement, the direction of improvement, the source of the data, the frequency of measurement and its owner.

Defining the measure properly also provides an opportunity for the BA to challenge the quality of the underlying information. A beautifully defined KPI is useless if the organisation cannot obtain the data needed to calculate it reliably.

A KPI Is Not a Target

KPIs and performance targets are closely related, but they are not the same thing. The KPI tells us what we are measuring; the direction tells us which way represents improvement; and the performance target tells us how good we need or want performance to be.

Suppose our KPI is the percentage of customer problems resolved on first contact. We establish that higher is better and discover that current performance is 71%. That still does not tell us what level the organisation considers satisfactory. A target might therefore be to achieve at least 85% within twelve months.

We now have a much richer statement of performance:

• CSF: customer problems are resolved effectively;
• KPI: percentage resolved on first contact;
• direction: higher is better;
• current performance: 71%;
• target: at least 85%;
• time horizon: twelve months.

Where Should Targets Come From?

A target should not be chosen simply because a number sounds impressive. Historical performance can provide evidence about what the organisation has achieved before and the rate at which performance has been changing. Benchmarking may show what comparable organisations, departments or teams achieve. Regulatory or contractual requirements may establish a minimum level that simply has to be met.

Targets may also reflect strategic ambition. An organisation seeking to differentiate itself through exceptional service may deliberately set a target above the industry norm. Risk tolerance provides another basis: even if an aspirational target has not been reached, management may need to define the point at which deteriorating performance becomes unacceptable.

Often several of these factors need to be considered together. Suppose an airline currently has a repeat-passenger rate of 42%. Comparable airlines average 47%, while the organisation's strategy is to establish a premium service with unusually strong customer loyalty. A target of 50% within eighteen months can now be discussed rationally. It may still be challenged, but it has a basis. That is very different from somebody simply announcing, "Let's make it 50%."

Targets, Thresholds and Tolerances

A single target value is not always enough because management may need to distinguish between several levels of performance. There may be a minimum acceptable level below which intervention is required, an expected level for normal operations, a target that the organisation is actively seeking, and perhaps a stretch level representing exceptional performance. Upper and lower tolerances can also be useful where success means remaining within a range.

Suppose the target complaint rate is no more than five complaints per thousand customers, while management establishes an escalation threshold of seven. The purpose of that threshold is not to redefine success. It provides an early warning that performance is moving far enough in the wrong direction to require attention. Used in this way, performance measurement becomes operationally useful rather than merely historical.

One Measure Can Distort the Picture

There is another danger in relying too heavily on a single KPI. Suppose management decides that customer calls take too long and introduces average call duration as a KPI, with lower values considered better. Staff respond exactly as the measurement system encourages them to respond: calls become shorter.

Unfortunately, customers begin calling back because their problems were not resolved. Repeat-contact rates increase and satisfaction falls. The organisation has successfully improved one KPI while damaging the CSF it was supposed to support.

A better measurement framework might therefore balance average response time with first-contact resolution, repeat-contact rate and customer satisfaction after resolution. Together they provide a more rounded view. This does not mean that every CSF should generate dozens of KPIs; too many measures simply recreate the original problem. The objective is to identify the smallest useful set of evidence needed to understand whether the critical outcome is being achieved.

Who Decides What Success Means?

Defining CSFs, KPIs and targets is a stakeholder exercise because different stakeholders see different aspects of performance. The department head may understand strategic priorities and what senior management expects, while operational managers understand the reality of day-to-day performance, bottlenecks and practical constraints. Finance can identify cost implications and the relationship between operational performance and financial outcomes, while risk and compliance specialists understand regulatory exposure, audit findings and organisational tolerance for particular failures.

Customers — internal or external — contribute another essential perspective because they can tell us what they actually value and what they experience as poor service. Data owners and IT specialists can establish whether the information required for a proposed KPI exists, whether it is reliable and whether measurement is practical.

These perspectives may conflict. That is not a reason to avoid the exercise; it is one of the reasons Business Analysis is needed.

What If the Organisation Says It Has No CSFs?

Business Analysts sometimes encounter the response, "We don't have Critical Success Factors." That usually means they have not been explicitly identified or documented. It rarely means that they do not exist.

Changing the questions can help. Ask what defines success, what must never go seriously wrong, what would cause senior management immediate concern, what customers complain about, where the department spends most of its money, or which failure would damage the organisation most. You might also ask what would make somebody question why the department exists at all.

Critical factors often become much easier to see when the discussion moves away from management terminology and towards consequences. A department may never have written down its CSFs, but it certainly has things that must go right if it is to remain successful.

From Measurement to Business Analysis

This is where CSFs and KPIs become much more than a management-reporting exercise. They can provide a direct connection between organisational objectives and the work undertaken by projects and change initiatives.

Suppose a project proposes a new customer-service platform containing automated case routing, a searchable knowledge base, customer history for service staff, automatic escalation of overdue cases and management reporting. All of those capabilities may sound reasonable, but the Business Analyst should still ask why they are needed.

If the relevant CSF is effective problem resolution and the current first-contact-resolution KPI is significantly below target, we have a much stronger basis for analysis. Will the knowledge base help staff resolve more enquiries without escalation? Will customer history prevent people having to repeat information? Will automated routing get specialist cases to the right person more quickly?

The requirements are no longer isolated features. They become proposed interventions intended to change measurable business performance.

Trace Requirements Back to Success

This gives the BA a powerful challenge when investigating requirements. For an important requirement or capability, ask which business objective it supports, which Critical Success Factor it affects, which KPI should change if it succeeds, which direction that KPI should move, what performance gap is being addressed and how confident we are that the proposed change will actually influence it.

Not every low-level requirement needs an elaborate individual business case; that would create bureaucracy rather than insight. At feature, capability or initiative level, however, the connection should be visible. If nobody can explain how a proposed change contributes to organisational success, the BA has found something worth challenging.

KPIs Can Help Build the Business Case

Performance gaps can also provide useful evidence for investment decisions. Suppose three departments are each requesting funding for improvement. One has a KPI slightly below an aspirational target, another has a KPI deteriorating rapidly towards an unacceptable risk threshold, while the third is comfortably achieving all its important targets but would like a more modern system.

That information does not automatically decide which project should be funded, but it makes the discussion considerably more useful. Initiatives can be compared in terms of the CSFs they affect, the size of the performance gap, the financial consequences of that gap, the risks associated with continued underperformance and the likely effect of the proposed intervention. This helps move prioritisation away from whoever makes the most persuasive presentation or has the most senior sponsor.

KPIs as an Early Warning System

Performance measures can also help an organisation act before a problem becomes a crisis. Suppose the target complaint rate is five or fewer complaints per thousand customers, but successive measurements show:

4.8  →  5.2  →  5.9  →  6.6

The target has already been missed, but waiting for catastrophic failure before responding would make little sense. If the organisation has established seven complaints per thousand as an escalation threshold, the trend provides an opportunity to investigate before that threshold is crossed.

The BA may become involved in identifying causes, analysing processes, investigating data, assessing options or defining an intervention. In this way, performance measurement does not simply tell management what went wrong last quarter; it helps identify where change may be needed next.

Do Not Forget Ownership

Every important KPI needs an owner. Ownership does not mean that one person is personally responsible for every factor affecting the number; it means somebody is responsible for ensuring that the measure is understood, reviewed and acted upon.

Without ownership, KPIs have a tendency to become permanent dashboard furniture. The number turns red every month, everybody notices and nobody does anything. Eventually the red indicator becomes normal. At that point the organisation is no longer managing performance; it is decorating reports with evidence of failure.

Keep the Framework Simple

The resulting framework does not need to become an enormous document. For many departments, a concise matrix may be enough, recording the Critical Success Factor, KPI, direction of improvement, current performance, target, warning or escalation threshold, measurement frequency, owner and the risk or consequence if the target is missed.

The purpose is not to produce another management artefact that everybody files away. The framework should remain sufficiently clear and visible to influence decisions.

Common Mistakes

Performance frameworks tend to go wrong in recognisable ways. One of the most common is confusing activity with success: counting how much work people perform does not necessarily tell us whether useful outcomes are being achieved. Another is having too many KPIs. If management receives fifty supposedly key indicators, they are not all key.

Other problems include ignoring direction, so that a changing number is reported without establishing whether the change is good or bad, and setting arbitrary targets with no defensible connection to evidence, strategy, regulation, benchmarking or risk. Organisations also frequently measure what is easy, allowing the availability of data to determine what receives management attention.

A single measure can be equally dangerous if people optimise it while damaging the outcome it was intended to represent. Finally, even an excellent KPI framework achieves little without ownership and action. Somebody needs to care when performance moves in the wrong direction, investigate why and decide what should happen next.

A Practical Exercise

You can apply this approach to almost any real department, service or business function. Choose one you know reasonably well and work through the following exercise:

1. Write its purpose in one sentence. Why does it exist and who depends upon what it provides?
2. Identify approximately five Critical Success Factors. Concentrate on outcomes rather than activities.
3. Define one or two KPIs for each CSF. Ask what evidence would genuinely tell you whether the factor was being achieved.
4. State the direction of improvement. Higher, lower, stable or within a range?
5. Establish current performance. Without a baseline, improvement cannot easily be demonstrated.
6. Propose performance targets. Explain where each target came from rather than simply choosing a number.
7. Identify warning or escalation thresholds. At what point should management intervene?
8. Identify ownership. Who reviews the measure and who acts when performance is unacceptable?
9. Identify the risks associated with failure. What happens if the CSF is not achieved?
10. Put the result onto one or two pages. If the framework becomes too complicated to use, simplify it.

When the exercise is complete, challenge every KPI with one final question: Why does this measure matter? If you cannot give a convincing answer, consider removing it.

The Business Analyst as Performance Architect

There is a broader lesson here about Business Analysis. Business Analysts are sometimes characterised primarily as people who gather and document requirements, but requirements are only one part of organisational change. A stronger chain looks like this:

Purpose  →  CSF  →  KPI  →  Performance Gap  →  Change  →  Requirements  →  Outcome

That chain allows the BA to ask not merely whether a requirement has been delivered, but why it existed in the first place and whether delivering it made the expected difference. After implementation, the organisation can return to the measures that justified the change. Did first-contact resolution increase? Did complaints fall? Did processing costs decrease? Did customer satisfaction improve? In short, did the change close the performance gap that justified the investment?

If the answer is no, that is valuable information. Perhaps the solution did not address the real cause, users did not adopt it, another constraint emerged, or the original assumption about what drove performance was wrong. The important point is that the organisation has something against which the outcome can be evaluated rather than simply declaring success because the project delivered its agreed scope.

What Should the Business Analyst Take Away?

Critical Success Factors, Key Performance Indicators and performance targets are sometimes treated as management terminology sitting outside the everyday work of Business Analysis. They should not. Used properly, they help answer some of the most important questions a BA can ask: why does this part of the organisation exist, what must go right for it to succeed, what evidence tells us whether that is happening, what level of performance do we need, where is the gap, what change might close it, and after the change has been delivered, did it actually work?

The discipline is therefore not really about dashboards, traffic lights or management reports. It is about connecting purpose, evidence, decisions and change. That connection gives the Business Analyst a way to move beyond documenting what people ask for and towards understanding what the organisation is trying to accomplish.