Your AI Passed Its Tests. Did You Test It for Bias?

An AI system can pass every test you planned for it. It can achieve its target accuracy, process the expected volumes, respond within the required time, survive its security testing and perform exactly as its developers expected. Yet it can still systematically disadvantage some of the people who use it.

That raises an uncomfortable question for anyone involved in delivering or testing AI systems. If nobody deliberately went looking for bias, what makes us think our testing would have found it?

Bias is often discussed as an ethical issue, and understandably so. An AI system that systematically treats people unfairly can have serious consequences, particularly when it influences decisions about employment, healthcare, education, insurance, credit or access to public services. However, treating bias solely as a matter for ethicists, lawyers or governance committees misses something important. For those of us concerned with the quality of systems, bias is also a testing problem.

It is an unusually difficult testing problem because bias does not necessarily result in obviously incorrect behaviour. The system may be doing exactly what it was trained to do. Indeed, the better it learns from its data, the more faithfully it may reproduce patterns that we would rather it had not learned at all.

Bias doesn't begin with the algorithm

Consider an organisation developing an AI system to identify employees who have the potential to become successful senior managers. It has ten years of personnel records containing performance assessments, promotion histories, qualifications, experience and other information about employees who progressed through the organisation. Training an AI model using the characteristics of people who were successfully promoted in the past might seem an entirely reasonable place to start.

Now suppose that, for historical reasons, most of those successful senior managers came from one demographic group. The dataset may be perfectly accurate. Nobody needs to have deliberately manipulated it, selected discriminatory data or programmed a prejudiced rule. Nevertheless, a model trained on those records may discover characteristics associated with historical promotion and use them as predictors of future potential.

The system has not malfunctioned. On the contrary, it may have learned its historical data extremely well. The difficulty is that it may also have learned yesterday's inequalities and converted them into tomorrow's recommendations.

This distinction matters because it changes how we think about testing. If we look only for software defects or obviously incorrect predictions, there may be nothing to find. We have to investigate not merely whether the system is working, but what it has learned and whether that produces materially different outcomes for different people.

There are many opportunities for bias to get in

It is tempting to think of AI bias primarily as a data-quality problem: find the biased data, clean it and retrain the model. Sometimes that is appropriate, but it seriously understates the problem. Bias can be introduced at almost every stage of an AI system's lifecycle, including stages that occur before anybody has assembled a training dataset.

It can begin with the definition of the problem itself. What does the organisation mean by a "successful employee", a "high-risk customer" or a "good candidate"? Who decided? Which outcomes are being optimised, and whose interests are represented by them? A poorly framed objective can shape everything that follows, however technically accomplished the eventual system might be.

Data collection creates further opportunities. Some populations may be poorly represented simply because fewer observations have historically been recorded for them. Data may have been collected using different methods in different locations or at different times. Historical records may contain the consequences of earlier human decisions, including decisions that would not meet today's expectations of fairness.

Labelling can introduce another layer of human judgement while disguising it as objective data. A field containing "successful applicant", for example, looks reassuringly factual until we ask how applicants historically came to be classified as successful. Similarly, concepts such as good performance, suspicious behaviour, creditworthiness and high potential may ultimately depend upon decisions made by people or institutions.

Removing protected characteristics does not necessarily solve the problem either. An organisation might carefully exclude sex, ethnicity or another sensitive attribute while retaining postcode, education, employment history or other variables that correlate strongly with it. The model does not need to be told explicitly which group somebody belongs to if other features allow it to reconstruct much of the same information.

Bias can also be affected by sampling, preprocessing, feature engineering, model optimisation and the thresholds chosen when converting model outputs into decisions. It can even emerge after deployment as populations and behaviour change, or as the system begins influencing the very environment from which its future data is collected.

There is therefore no single point in an AI project at which a team can confidently tick a box marked "bias checked" and move on.

The average can hide a great deal

This creates a particular challenge for testers because many of the measures routinely used to demonstrate AI performance are aggregate measures. A model that achieves 94 per cent accuracy may sound impressive, and in some circumstances it may indeed be impressive. But that figure tells us how the model performs across the population we used to calculate it. It does not necessarily tell us how well everybody within that population is being served.

Suppose we divide those results into meaningful groups and discover that the model achieves 96 per cent accuracy for one population but only 76 per cent for another. The original 94 per cent has not become mathematically incorrect; it has simply concealed information that matters.

Accuracy alone may not reveal the most important disparity anyway. We might discover that overall performance is broadly comparable while false-positive rates are substantially higher for one group. In another application, false negatives may be the critical issue. A medical diagnostic system, for example, could appear to perform well overall while being appreciably more likely to miss a condition in one population. A fraud-detection system could identify genuine fraud effectively while also subjecting customers from a particular group to disproportionately frequent false alarms.

The aggregate metric has not lied to us. We simply asked it a question that was too broad.

That is one of the most useful habits testers can bring to AI: whenever presented with an impressive overall performance figure, ask what happens when the population is divided in ways that might matter.

Testers need to ask a different kind of question

Traditional testing asks: “When might the system give the wrong answer?”

Bias testing adds: “For whom might it systematically give a different answer?”

Much conventional testing begins with variations of the same underlying question: under what circumstances might this system give the wrong answer? We construct boundaries, equivalence partitions, unusual combinations, error conditions and other tests intended to expose those circumstances.

Bias testing requires us to add another question: for whom might this system systematically give a different answer?

Once we begin asking that question, familiar testing techniques become surprisingly useful. Instead of partitioning only by input ranges or system states, we can partition outcomes by relevant population characteristics. We can compare error rates, investigate boundaries, construct carefully controlled cases and explore combinations that may not have been adequately represented in the original data.

There is then a harder question: is the difference justified?

This is where bias testing becomes more complicated than simply searching for disparities. Different outcomes do not automatically demonstrate unfairness. There may be legitimate reasons why risk, treatment or recommendations differ between populations or circumstances. Equally, a difference that can be explained statistically is not automatically ethically, legally or commercially acceptable.

A tester can expose the difference and provide evidence about its extent and behaviour. Deciding whether it is justified may require domain specialists, affected stakeholders, governance and risk functions, legal expertise and ultimately those responsible for accepting the system into operation. The tester's responsibility is not necessarily to pronounce a system fair or unfair; it is to ensure that important differences do not remain hidden inside reassuring averages.

What would testing for bias actually involve?

There is no single test that establishes that an AI system is unbiased, which is why teams need a systematic approach rather than a final "fairness test" shortly before release.

Investigation should begin with the data. Which populations are represented and in what numbers? Are there groups for which there are so few examples that claims about model performance become unreliable? Does the dataset represent the population in which the system will actually operate, and are important cases absent because they were uncommon or historically overlooked?

The provenance of labels deserves similar attention. Teams should understand whether a label represents an objective observation or somebody's judgement and, where judgement was involved, how consistently it was applied. Historical labels deserve particular scrutiny because a model trained to reproduce previous decisions may also reproduce the assumptions behind them.

Test results should then be partitioned rather than examined solely at population level. Accuracy, precision, recall and error rates can be compared across relevant groups, with false positives and false negatives examined separately where their consequences differ. Which partitions are appropriate will depend upon the system and its context; mechanically dividing every dataset according to every conceivable characteristic is no substitute for understanding where unfairness could plausibly arise.

Intersectionality makes this more challenging. Testing separately by age and sex, for example, might suggest no significant problem while a particular combination — perhaps older women — experiences noticeably poorer outcomes. The more finely we partition a population, however, the smaller our samples become, so apparent differences must also be treated with appropriate statistical caution.

Counterfactual testing provides another useful technique. If two otherwise equivalent cases differ only in a characteristic that should have no bearing on the decision, do they receive materially different outcomes? Real systems make constructing genuinely equivalent cases difficult, but the principle is valuable: change something that should not matter and observe whether it does.

None of this should stop when the system is released. Operational populations change, behaviour changes and data drifts. Feedback loops may develop as AI-generated decisions influence subsequent human behaviour and future training data. Bias testing therefore belongs within monitoring and periodic reassessment just as much as it belongs within pre-release evaluation.

Mitigation is not the end of testing

Finding a disparity naturally leads to the question of how it should be corrected. Depending upon its cause, mitigation might involve collecting better data, changing sampling or weighting, reconsidering features, modifying labels, adjusting thresholds, changing the model or introducing human review into particular decisions.

The danger is treating mitigation as a repair that simply removes the problem. Fairness frequently involves trade-offs. A change that reduces false negatives may increase false positives. A technique that improves one measure of fairness may worsen another. An intervention intended to improve outcomes for one group may have unexpected consequences elsewhere in the population.

For that reason, mitigation should be treated in exactly the way testers would treat any other significant change to a system: make the change, understand what it was intended to achieve, identify its possible side effects and test again.

There is no "make unbiased" button. More importantly, there is rarely a point at which a responsible project team can prove that no form of bias exists anywhere in a complex AI system. What it can do is demonstrate that credible sources of bias have been investigated, identified disparities have been examined, mitigations have been tested and residual risks are understood.

Bias is a project responsibility

This is also why responsibility for bias cannot sensibly be handed to a single specialist. An AI ethicist may provide valuable challenge, but cannot compensate for a project team that has never considered bias in its everyday decisions. Nor can the tester simply be given a model at the end of development and asked to determine whether it is fair.

Business Analysts can challenge how the original problem and requirements have been framed. Domain specialists can question whether labels and assumptions really mean what developers think they mean. Data scientists can investigate representation, features and model behaviour. Testers can design experiments specifically intended to reveal disparities. Risk, legal and governance specialists can challenge the evidence and implications, while product owners and senior decision-makers ultimately have to understand and accept whatever residual risk remains.

Each sees a different part of the problem. More importantly, each has opportunities either to prevent bias or inadvertently to introduce it.

That suggests that organisations need to think about bias in much the same way that mature development teams eventually learned to think about security and quality. It is much harder to inspect either of them into a finished product than to consider them throughout its development.

Go looking for it

AI has changed one of the assumptions behind testing. With conventional software, we devote considerable effort to finding circumstances in which the system fails. With AI, we must also consider the possibility that the system appears to be working perfectly while repeatedly producing poorer outcomes for particular people.

Those differences can disappear inside an excellent average. They can be inherited from history, introduced through apparently reasonable project decisions or emerge only after the system encounters the real world. None of them necessarily produces an error message, a failed test case or an obvious defect.

That is why bias has to be something we deliberately look for.

Passing the planned tests tells us something important about an AI system, but it cannot tell us whether we asked all the questions that mattered. Before declaring the testing complete, perhaps every AI project should therefore ask one final question: