AI Red Teaming

If your testing follows the rules, it won’t find real failures

In recent years, “red teaming” has moved from being a niche concept associated with cyber-security and military exercises into the mainstream of AI governance. That shift has not happened by accident. As AI systems have become more capable, more autonomous, and more deeply embedded in critical business and societal functions, regulators and organisations have begun to recognise a difficult truth: conventional testing alone is no longer sufficient.

Modern AI systems do not merely execute predefined logic. They respond probabilistically, adapt to context, and interact with users in ways that are often unpredictable. A system may perform flawlessly within a controlled testing environment yet behave very differently once exposed to the ambiguity and pressure of real-world use. In that sense, AI has forced the software industry to confront a problem long understood in other operational domains – namely, that systems must be tested not only for correctness, but for resilience under adversarial conditions.

This growing concern is reflected in the EU Artificial Intelligence Act, which has significantly accelerated interest in formal adversarial evaluation. Contrary to some headlines, however, the Act does not simply state that “all AI systems must undergo red teaming.” The reality is more nuanced. The legislation adopts a risk-based approach, imposing its strongest obligations on a particular category of advanced models known as General Purpose AI (GPAI) systems with systemic risk. In practical terms, this refers to large, highly capable foundation models whose scale, reach, and potential impact mean that failures could have broad societal consequences. Ordinary AI systems used in relatively constrained business contexts do not face the same level of scrutiny.

Frontier-scale models, however, are increasingly expected to undergo extensive evaluation, including adversarial testing, model assessment, risk mitigation, monitoring, and incident reporting.

This distinction between ordinary AI and systemic-risk GPAI is important because it reflects a wider shift in regulatory thinking. Traditional software assurance has tended to focus on whether systems behave correctly under expected conditions. The EU AI Act moves beyond that model by recognising that advanced AI systems may exhibit unexpected or emergent behaviours that cannot be fully understood through conventional validation alone. As a result, organisations developing or deploying these powerful models are being pushed toward a much more rigorous approach to assurance – one that includes deliberately attempting to provoke failure, expose weaknesses, and identify harmful or unsafe behaviours before systems are widely deployed.

Within this context, “red teaming” takes on a meaning that is broader and more disciplined than the popular image often presented online. In the EU context, adversarial testing is not simply a matter of producing amusing jailbreak prompts or isolated examples of model failure. It is increasingly understood as a structured process of evaluating systemic risk. That includes probing for:

• robustness failures
• unsafe outputs
• manipulation
• misuse potential
• harmful emergent behaviour
• security weaknesses
• the ways in which a system behaves under sustained pressure or ambiguous conditions.

Importantly, regulators are still in the process of defining exactly what “good” looks like in this space. The AI Act establishes obligations and expectations, but many of the practical standards, methodologies, and evidential requirements are still evolving through codes of practice, technical guidance, and industry consensus. In effect, one might say that regulators are formalising the principle of adversarial evaluation before the discipline itself has fully matured.

For software testers, this matters enormously. It signals a fundamental shift in what assurance is expected to mean in the age of AI. Organisations are no longer being asked simply to demonstrate that systems work; increasingly, they are expected to demonstrate that systems have been challenged, stressed, and examined under conditions that deliberately attempt to make them fail. That changes the role of testing itself. It introduces questions of independence, adversarial thinking, systemic risk, and organisational willingness to confront uncomfortable findings. In many respects, the challenge facing AI assurance today is not technological but cultural. Systems that appear robust under controlled conditions may still contain weaknesses that only emerge when assumptions are broken and pressure is applied.

So what, then, do we actually mean by “Red Teaming”?

There is a growing confidence in the way AI systems are being tested.

You see it in demonstrations, in assurance reports, and in the increasing use of the phrase “red teaming” to describe how systems are being challenged. It creates the impression that testing has evolved – that we are no longer simply validating behaviour but actively probing for weaknesses.

On the surface, that confidence feels justified – but there is a problem.

Much of what is currently described as red teaming still operates within the same mental model as conventional testing. It is structured, contained, and – perhaps most importantly – polite. It challenges the system, but it does so within an unspoken agreement about how that challenge will be conducted. And that agreement fundamentally undermines the exercise because real adversarial testing does not respect those boundaries.

If that sounds abstract, it helps to look at an environment where this distinction has already been confronted – and resolved – not in theory, but in practice.

Testing Under Assumption

In most software environments, testing begins with an assumption: that the system can be evaluated by defining expected behaviour and verifying that it occurs.

This assumption is rarely questioned, because it works well in deterministic systems.

We define inputs, we predict outputs, we verify that the system behaves as intended. Even when we extend into negative testing or exploratory approaches, the structure remains the same. The system is still being asked to demonstrate that it works, albeit under slightly more varied conditions.

There is an implicit contract in this process. The tester defines the space in which the system will be evaluated, and the system is judged based on how well it performs within that space. This is entirely reasonable – provided that the defined space reflects reality.

The Moment Assumptions Break

There are systems – operational systems, not theoretical ones – where this model of testing proves insufficient.

In these environments, it is not enough to demonstrate that something works under expected conditions. What matters is how it behaves when those conditions no longer apply.

That is where adversarial evaluation begins and it is here that the distinction between testing and being tested becomes important.

In conventional testing, we control the environment but in adversarial testing, the environment is allowed to control us.

That inversion is not comfortable. It removes the certainty that comes from defined scenarios and predictable outcomes. It introduces ambiguity, unpredictability, and, at times, failure but it also produces something that conventional testing cannot: a realistic understanding of how a system behaves under pressure.

A Practical Illustration

All military units in the United Kingdom have their operational readiness regularly assessed through commonly-used exercises known as MINEVALs. At longer periods, external specialists arrive to conduct serious drills lasting days.

These are not theoretical drills. They are designed to test how the unit would respond to a genuine threat and, to test that properly, they are unannounced. The first indication that an exercise is underway is often the sirens sounding in the early hours – 02:00 is not unusual – pulling people out of sleep and into action. There is no preparation window and no controlled setup.

The expectation is simple: respond as you would if the situation were real.

Minevals are arranged by the unit itself. That means they are, by definition, constrained by internal assumptions. The scenarios are designed to test known procedures, known responses, and known threat models – the tactical playbook, in other words.

They are useful. They confirm readiness – but they also contain an inherent limitation.

They test the system within the boundaries that the system itself understands.

Introducing the Adversary

On one occasion at a particular RAF station, that limitation was deliberately challenged.

Instead of designing the exercise internally, the decision was made to introduce an external adversary. A territorial SAS unit was brought in to act as the attacking force. This was no symbolic gesture. It was a deliberate attempt to introduce a different mindset – one not bound by the same assumptions as those being tested.

Their captain was invited to the station in advance. He was not given a script – he was given visibility. He observed. He assessed. He thought.

When he left, the mood across the station was confident. The defences were strong. The procedures were sound. The expectation was that whatever approach the attackers took, it would be detected and handled. That confidence was not unreasonable – those serving at the station were professional, motivated and well-trained (just unaware of the impending exercise).

But that confidence was also misplaced.

When the Exercise Became Real

In the early hours of the exercise, the unexpected happened – suddenly, vehicles appeared on the runways. They moved quickly, aggressively, attacking from positions that should not have been accessible. Defensive responses were triggered, but there was confusion. The perimeter had not been breached – at least, not in any way that had been anticipated.

The question was not whether the station could respond. It was how the attackers had got there in the first place. And the answer, when it came, did not emerge immediately.

It came later.

The Hedge

Part of the station perimeter was delineated by a large, dense yew hedge – thick, mature, and, by all accounts, secure. It was not treated as a primary point of vulnerability. It was simply a treasured part of the environment. Elsewhere, constant, alert patrols ensured no known permeability existed.

On the night of the captain’s visit, many hours after he had left, that assumption was quietly dismantled.

His team arrived and cut out a 2-metre section of the hedge – not “hacked through” and not damaged in a way that would attract attention. It was very carefully removed.

Vehicles were then driven through the opening and positioned within the perimeter, in a disused area where they would not be noticed. Then the hedge section was carefully pushed back in – invisible mending at its best – meaning that from the outside, nothing had changed.

When the exercise began, the attackers were already inside.

Breaking the Model

This was not how the exercise was supposed to work.

It did not follow the expected pattern of attack. It did not engage with the perimeter in the way the defenders had anticipated. It bypassed the entire model on which the defensive strategy had been built. It was, in a sense, unfair. In fact, the station only discovered how they had gained entry two weeks later when that section of the hedge turned yellow!

It was an entirely effective tactic. What it revealed was not a failure of execution, but a failure of assumption under stressful conditions.

The defenders had prepared for a particular kind of threat, yet the attackers chose another entirely, and in doing so the exercise revealed not only the level of capability in handling the attacks, but the extent to which that capability had been shaped – and constrained – by its underlying assumptions.

Why This Matters for AI

This distinction – the difference between testing a system and testing the assumptions around it – is directly relevant to AI.

Most AI testing today operates within a defined model of behaviour. We construct datasets that represent expected use. We define scenarios that reflect anticipated interactions. We measure performance within those boundaries and, within those boundaries, systems perform well but those boundaries are not reality.

Real-world interaction with AI systems is messy, inconsistent, and often adversarial – whether intentionally or not – because inputs are seldom neatly structured, context evolves, and users persist, probe, and adapt as the system responds.

In short, the system is not used in the way it was tested.

The Illusion of Confidence

This creates a familiar pattern:

• A system is tested thoroughly within a defined space. It performs well. Confidence builds.
• Then the system is deployed into an environment that does not respect that space.
• Behaviour begins to diverge.
• Not catastrophically, perhaps – nor immediately – but gradually, inconsistently, and often in ways that are difficult to predict.

At that point, the question is no longer whether the system works. It is whether we understand how it fails.

And that is a question that conventional testing is not well equipped to answer.

Where AI Red Teaming Falls Short

If the lesson from adversarial evaluation is that systems must be tested outside their own assumptions, then it follows that the effectiveness of AI red teaming depends entirely on how far it is willing to move beyond those assumptions.

At present, much of it does not move very far.

This is not due to a lack of effort. It is due to a persistence of the same mindset that governs conventional testing: the belief that the testing space can be defined, controlled, and meaningfully explored within boundaries that feel reasonable.

That belief is precisely what adversarial testing is meant to challenge.

The Persistence of the “Test Case” Mindset

One of the clearest signs of this limitation is the continued reliance on test cases – even when conducting what is described as adversarial testing.

The structure is familiar: a scenario is defined, an input is constructed and then system’s response is observed.

In AI contexts, this often takes the form of prompt design. A tester attempts to craft a prompt that will cause the system to behave in an undesirable way. If successful, the prompt is recorded as evidence of a vulnerability.

This approach has merit – it demonstrates that certain behaviours are possible, but it retains the underlying assumption that the testing space can be explored through discrete, isolated examples.

That assumption does not hold.

AI systems do not fail in single moments. They fail across patterns – patterns of input, of context, of interaction over time. A single prompt, no matter how effective, tells you very little about the system as a whole.

It is the equivalent of identifying a single weak point in a perimeter without understanding whether it is guarded, how often it is exposed, or whether it can be exploited repeatedly.

The Problem of “Polite” Adversaries

There is another, subtler limitation.

Even when testers attempt to adopt an adversarial stance, they often remain constrained by an implicit sense of what is acceptable. Inputs are constructed with care, the language is kept under control, and the interaction itself tends to follow a familiar, recognisable pattern. In other words, the tester behaves like a cooperative participant attempting to be slightly difficult.

This is not how adversarial behaviour manifests in the real world.

Real users – whether malicious, careless, or simply persistent – do not follow structured patterns. They may repeat themselves or contradict what they have already said, adjusting their approach in response to the system and often probing its boundaries indirectly, sometimes without even realising it.

A testing approach that remains “polite” cannot replicate this behaviour. It produces a sanitised version of adversity – one that may reveal obvious weaknesses but is unlikely to expose deeper ones.

The Focus on Novelty Over Understanding

Another common feature of current red teaming practice is a focus on novelty.

There is a natural tendency to highlight clever or unexpected ways of provoking undesirable behaviour. These examples can be engaging and often show real ingenuity, which naturally draws attention, but they can also distract from the underlying objective.

The purpose of adversarial testing is not to find the most surprising failure. It is to understand how failures occur, how they behave, and what they imply about the system. A novel example may demonstrate that a system can be broken. It does not, on its own, provide insight into the likelihood or impact of that failure.

Without further exploration, it remains a curiosity.

The Absence of Pressure

Perhaps the most significant limitation is the absence of sustained pressure.

In many red teaming exercises, interactions are brief. A prompt is issued, a response is observed, and the exercise moves on.

This approach assumes that failures are immediate – that they will reveal themselves in a single exchange but, in practice, this is rarely the case.

Complex systems often exhibit weaknesses only under continued stress. Behaviour may appear stable in isolation but begin to degrade as interactions accumulate. Context may introduce subtle shifts that only become visible over time. We often see this in LLMs when the application starts to hallucinate as the context gradually reaches full capacity. Without sustained pressure, these behaviours remain hidden.

The system is tested but not truly challenged.

The Result: Demonstration Without Depth

Taken together, these limitations produce a form of testing that is effective at demonstrating that failures exist, but less effective at explaining them.

The system is shown to be vulnerable, but the nature of that vulnerability remains unclear. This is the equivalent of discovering that a perimeter can be breached, without understanding how, how often, or with what consequences.

It is a starting point and not an end state.

If adversarial testing is to move beyond demonstration, it must adopt a different set of practices – ones that reflect the complexity and variability of the systems being tested. This does not require abandoning structure.

It requires redefining it.

From Test Cases to Test Strategies

The first shift is away from individual test cases and toward test strategies.

Instead of asking, “What input should I use?”, the question becomes, “What kind of pressure am I trying to apply?”

This leads to a different way of thinking about testing.

A strategy might involve:

• exploring ambiguity
• introducing contradiction
• extending context over multiple interactions
• probing boundaries incrementally

Each of these strategies can generate multiple test paths. The objective is not to find a single successful input, but to understand how the system behaves under a particular form of stress.

The Importance of Variation

Variation is central to this approach.

A single input is of limited value. What matters is how the system responds to changes – small changes in phrasing, structure, or context.

By introducing variation, the tester can begin to identify patterns:

• Does the failure persist?
• Is it sensitive to specific wording?
• Does it generalise across similar inputs?

This moves the exercise from possibility to probability.

Extending Interaction Over Time

Many AI systems, particularly conversational ones, maintain context across interactions. This creates an opportunity – and a requirement – for testing that extends beyond single exchanges.

A disciplined adversarial approach will:

• build on previous responses
• introduce new elements gradually
• observe how the system adapts over time

In doing so, it can reveal behaviours that are not visible in isolated interactions. A system may appear stable in a single response but become inconsistent or unreliable as context accumulates.

Contradiction is a powerful tool in adversarial testing.

By presenting the system with conflicting instructions or information, the tester can observe how it resolves ambiguity.

• Does it prioritise one instruction over another?
• Does it attempt to reconcile the conflict?
• Does it produce inconsistent or incoherent output?

These behaviours provide insight into the system’s underlying decision-making processes.

A Worked Example

Consider a system designed to provide structured advice within a defined domain.

A conventional test might involve asking a clear, well-formed question and verifying the response. An adversarial approach would begin differently.

The tester might start with a vague or ambiguous question, observing how the system interprets it. Then the question is refined, but in a way that introduces subtle inconsistencies. The system’s response is compared.

Next, the tester introduces a previous response into the conversation, creating a chain of interaction. The system is asked to build on its own output.

Finally, contradictory instructions are introduced.

At each stage, the tester is not simply looking for a failure, but for a pattern.

• Does the system remain consistent?
• Does it drift?
• Does it reveal weaknesses in how it handles context or ambiguity?

By the end of the exercise, the tester has a much richer understanding of the system’s behaviour.

From Failure to Risk

The final step is to translate observed behaviour into risk.

Not all failures are equal. Some are rare, difficult to reproduce, or of limited impact. Others are frequent, easily triggered, and potentially significant. A disciplined red teaming exercise must distinguish between these.

This requires:

• repetition to assess frequency
• variation to assess robustness
• analysis to assess impact

Only then can findings be prioritised and acted upon.

Why Organisations Struggle With This

If the principles of effective adversarial testing are clear, why are they not more widely applied?

The answer lies less in technique than in culture - adversarial testing exposes weaknesses.

This is its purpose, but it also makes it uncomfortable. It challenges confidence in systems that may have been presented as robust. It produces results that are not always easy to explain or resolve and there is a natural human tendency to limit this exposure.

Testing is constrained, while scenarios are defined resulting in the exercise being kept within manageable bounds and, in doing so, the value of the exercise is reduced.

The Preference for Predictability

Organisations tend to favour processes that are predictable and controllable.

Adversarial testing is neither.

It introduces uncertainty. It may produce unexpected results. It may challenge assumptions that are deeply embedded in the system’s design. This makes it harder to integrate into existing processes.

There is also a concern that findings from adversarial testing may be misunderstood.

A failure may be interpreted as a flaw in the system, rather than as an expected outcome of a challenging test. Without careful communication, this can lead to resistance.

As a result, testing may be softened – intentionally or otherwise – to avoid producing results that are difficult to manage.

Closing Perspective

AI systems are being deployed into environments that are complex, dynamic, and often unpredictable. Testing that assumes stability will not be sufficient.

Where this is likely heading over the next few years is becoming increasingly clear. Red teaming is unlikely to remain an informal or loosely defined activity conducted at the margins of AI development. As regulatory expectations harden, adversarial evaluation will almost certainly become more structured, more evidence-driven, and more closely tied to governance and assurance frameworks.

We are already seeing the early signs of this transition: the emergence of formal model evaluations, growing pressure for independent assessment, and increasing recognition that frontier AI systems cannot be treated in the same way as conventional software. In time, it is likely that routine validation and adversarial evaluation will become recognised as distinct disciplines, each serving a different but complementary purpose.

At the same time, organisations may discover that the greatest challenge is not technical implementation but institutional mindset. True adversarial testing is uncomfortable because it exposes assumptions, challenges confidence, and reveals weaknesses that may not have straightforward solutions. There will always be a temptation to constrain the exercise, soften the findings, or reduce red teaming to a compliance activity that demonstrates diligence without genuinely challenging the system. Yet the history of operational evaluation suggests that this is precisely the point at which assurance becomes unreliable.

A system that has only been tested within expected boundaries has not really been tested at all.

The EU AI Act, despite the uncertainty that still surrounds aspects of its implementation, signals something historically significant. Regulators are beginning to acknowledge that highly capable AI systems must not merely be shown to function; they must be subjected to disciplined attempts to make them fail. That represents a major philosophical shift in software assurance. It moves testing away from the narrow question of whether a system behaves correctly under anticipated conditions and toward the much harder question of how that system behaves when assumptions collapse, pressure accumulates, and adversarial behaviour enters the picture.

Ultimately, that is the real value of red teaming. It is not an exercise in cleverness, nor an attempt to embarrass developers or expose isolated flaws for their own sake. Its purpose is to force systems beyond the comfort of their own assumptions and, in doing so, reveal the difference between apparent robustness and genuine resilience.

In an era where AI systems are increasingly embedded into critical decision-making and operational environments, that distinction matters more than ever.