Who Gets to Say No?

The Emerging Role of the AI Ethicist

Imagine an AI project that has gone well. The system works, it meets its performance targets, the project is within budget, the lawyers are satisfied that its proposed use is lawful, the security team has signed it off, and the organisation has complied with the relevant regulations and governance procedures. By all the conventional measures available to the organisation, the project is ready to proceed.

And yet deploying it would be wrong.

Who, exactly, is expected to say so, and perhaps more importantly, who has the authority to make anyone listen?

As artificial intelligence moves from experimentation into ordinary organisational life, I suspect that question is going to become increasingly important. We already talk a great deal about AI ethics, and organisations routinely publish principles concerning fairness, transparency, accountability and human oversight. We talk rather less about the person, or perhaps the professional role, responsible for ensuring that those principles actually influence what an organisation does when commercial, operational and ethical considerations begin to pull in different directions.

Legal, compliant and ethical are not the same thing

There is an understandable tendency to bundle these concepts together. If an AI application is lawful, complies with regulation, has passed its risk assessments and performs as specified, there is a temptation to conclude that all the necessary questions have been answered. They haven't, because each of those activities is answering a subtly different question.

A lawyer can advise whether an organisation may legally do something. A compliance specialist can determine whether prescribed obligations have been met, while a risk specialist can identify and assess particular threats and consequences. Developers and testers can provide evidence about how the system behaves and whether it satisfies the requirements that have been placed upon it. All of those functions are necessary, but none of them necessarily answers the more fundamental question of whether the organisation should be doing this at all.

Sometimes the question becomes even more difficult. We may decide that there is a legitimate reason for using AI, but still have to ask whether we should use it in this particular way. Those are not simply technical or legal questions. They require judgements about people, fairness, proportionality, autonomy, harm and the distribution of benefits and consequences. If those judgements matter, then someone needs both the responsibility and the opportunity to make them.

What would an AI ethicist actually do?

I don't envisage an AI ethicist sitting at the edge of a project, occasionally being invited to a meeting to remind everyone that fairness is important. If the role is to have any real value, ethical challenge has to become part of the decision-making process itself and must begin early enough to influence what is being built rather than merely comment upon what has already been built.

An ethicist might therefore ask whether AI is an appropriate solution to the problem in the first place, who will benefit from the proposed system and who will bear its risks. They might ask which people affected by its decisions have no representation within the project, whether apparently innocent data items are acting as proxies for something much more sensitive, and whether a person can understand and challenge a decision that materially affects them. They would also need to consider what happens to those for whom the system fails, because averages and aggregate performance measures have a habit of concealing the experience of individuals.

Perhaps most importantly, the ethicist should be able to challenge the assumption that too easily becomes embedded once a technically exciting project gets under way. Just because we can build something does not necessarily mean that we should build it. That sounds obvious when expressed in the abstract, but it becomes considerably less obvious when an organisation has identified a significant commercial advantage, a large cost saving or a politically attractive opportunity to improve efficiency.

That is why the question needs to be asked early.

Ethics cannot be bolted on at the end

There is a parallel here with something the software industry learned, or at least ought to have learned, many years ago. You cannot test quality into a finished product. If meaningful testing begins only when development has finished, many of the important decisions have already been made. Defects become more expensive to correct, architectural choices become difficult to reverse and, inevitably, there is increasing pressure to accept things that would have been challenged much more vigorously six months earlier.

Ethics has much the same problem. Imagine introducing a serious ethical review only after an organisation has spent several million pounds developing an AI system. Contracts have been signed, senior executives have announced the initiative, delivery targets have been agreed and careers may now depend upon its success. Even without anybody behaving dishonestly, everyone involved has acquired a powerful incentive to discover reasons why deployment is ethically acceptable. The question has ceased to be entirely neutral.

The useful ethicist therefore needs to appear much earlier, perhaps at the moment somebody first says, "We could use AI to..." At that stage it is still possible to question the purpose, consider alternatives, identify affected groups and decide that a proposed application should be modified or perhaps should not proceed at all. Ethical thinking then becomes part of designing the system rather than an obstacle placed in front of it immediately before deployment.

But does the ethicist get a veto?

This is where the idea becomes more uncomfortable. An ethicist who can offer advice that everybody is free to ignore may amount to little more than ethical decoration, but giving one individual an absolute veto over major organisational decisions creates problems of its own. Ethical questions are rarely binary, and reasonable, well-informed people can reach different conclusions when genuine benefits and genuine harms have to be balanced against one another.

Perhaps, therefore, the answer is not an ethical veto but an ethical hold-point. At significant stages of an AI project, an independent ethical assessment might formally conclude that there is no material ethical objection, that the project may proceed subject to specified conditions, that material ethical concerns remain, or that senior escalation is required before the project can continue. The important thing is not necessarily that the ethicist has the final word, but that the ethical judgement becomes an identifiable part of the decision.

An organisation could still decide to proceed despite an ethical objection. There may even be circumstances in which it has perfectly defensible reasons for doing so. However, somebody with sufficient authority would then have to make that decision explicitly, explain why the anticipated benefits justify proceeding and, crucially, own the consequences of that decision.

That is very different from recording in the minutes that "ethical implications were discussed" and then moving on to the next agenda item. It creates an audit trail not merely of the fact that ethics was considered, but of what concerns were raised, what was done about them and who ultimately accepted responsibility for proceeding.

Independence matters

We have already accepted this principle elsewhere in organisational life. Independent testers are valuable partly because they are not the people who built the system and therefore bring a different perspective to claims that it works. Security specialists are expected to challenge designs that create unacceptable vulnerabilities, while safety professionals may have obligations that conflict directly with cost and schedule. Data protection roles can carry responsibilities that require a degree of independence from operational management precisely because the organisation's immediate commercial interests and its wider obligations will not always point in the same direction.

It is difficult to see why ethical challenge should be fundamentally different. The ethicist does not need to run the project, nor should every personal concern raised by an ethicist automatically override the judgement of everyone else involved. What matters is that ethical concerns have a recognised route into the governance structure and cannot simply disappear because they are inconvenient.

An AI ethicist without independence is little more than an adviser, while an ethicist without access to the people making the decisions risks becoming a commentator on decisions already made. Most seriously, an ethicist who has no meaningful ability to challenge deployment risks becoming decoration: evidence that the organisation has "done ethics" without giving ethical reasoning any real influence over the outcome.

What sort of person could do this job?

That raises another problem, because at present the title AI ethicist can mean almost anything. Is this person a philosopher, a lawyer, a sociologist, a technologist, a governance specialist or some combination of all of them? If organisations eventually come to regard AI ethics as a genuine professional discipline, the people performing the role will probably require an unusual mixture of skills.

They will need ethical reasoning, certainly, but understanding philosophy alone will not be enough. They will need sufficient understanding of AI and data to challenge technical claims intelligently without necessarily becoming data scientists themselves. They will need to understand governance, risk, regulation and organisational behaviour, because an ethical recommendation that cannot operate within the realities of an organisation is unlikely to achieve very much. They will also need to understand how apparently technical decisions about data, thresholds, optimisation and automation can have consequences that are anything but technical for the people subjected to them.

Perhaps most importantly, they will need something considerably harder to teach: the confidence and credibility to challenge senior people constructively. The most difficult ethical questions will rarely arrive labelled as ethical problems, and they certainly will not normally be as straightforward as asking whether an organisation should build an AI system that deliberately discriminates against people. Nobody needs an ethicist to provide the expected answer to a question framed that obviously.

The real questions will be much less comfortable. Suppose, for example, that a system will significantly improve the service received by 98 per cent of an organisation's customers, while the organisation knows that the remaining 2 per cent will be disadvantaged. Does the benefit to the overwhelming majority justify the harm to the minority?

An ethicist cannot answer that simply by pointing to the percentages. Someone has to ask who those 2 per cent are, whether they are always the same people, how seriously they are disadvantaged, whether an alternative approach exists and whether the disadvantage can be mitigated. We may also need to know whether those affected can appeal against the decision, whether a human being can intervene and whether the people concerned even know that an AI system was involved in producing the outcome. There may be no perfect answer, but that is precisely why the question matters.

Testing can tell us something—but not everything

My own background is in software testing, and AI has already forced testers to reconsider some comfortable assumptions about what it means for a system to work. With conventional software we are accustomed to asking whether the system produced the expected result. AI complicates that considerably, particularly where outputs are probabilistic and behaviour depends upon data as well as code. It becomes much harder to point to a single expected result and declare everything else to be a defect.

Ethics introduces another dimension. Testing can ask whether the system behaves acceptably, but ethics then has to ask, acceptably to whom? Once we ask that question, another follows naturally: who gets to define what "acceptable" means? Neither question can be answered satisfactorily by looking only at an accuracy figure on a test report.

Suppose an AI system achieves 95 per cent accuracy. That sounds impressive until we ask which 5 per cent it gets wrong and what being wrong actually means. If the errors are distributed randomly and their consequences are trivial, 95 per cent might represent excellent performance. If most of those errors affect one particular group of people and result in them being denied employment, insurance, medical treatment or access to a public service, exactly the same performance figure begins to look very different.

The metric has not changed. Our understanding of what the metric means has.

This is one of the reasons why I think the relationship between AI testing and AI ethics will become increasingly important. Testers can provide evidence about behaviour, consistency, bias, robustness and failure. The ethicist can challenge what those findings mean for the people affected and whether the residual harms are acceptable. Neither discipline replaces the other, but together they can expose questions that a simple statement that "the system passed testing" never could.

From ethical principles to ethical responsibility

Most organisations working seriously with AI will already have encountered principles such as fairness, transparency, accountability, privacy and human oversight. There is no shortage of ethical frameworks, and producing another list of desirable principles is unlikely, by itself, to change very much. Principles are useful because they provide a language in which ethical concerns can be expressed, but principles do not attend project meetings, challenge assumptions or make deployment decisions. People do.

As AI becomes embedded in increasingly consequential organisational processes, I think we may see ethics follow a path already travelled by security, safety, data protection and testing. What begins as a general responsibility eventually becomes sufficiently important and sufficiently specialised that organisations create identifiable roles, competencies, responsibilities and lines of authority around it. Once that happens, it will no longer be sufficient to ask whether an AI project has "considered ethics", because almost any project can answer yes to such a loosely framed question.

The better questions will be much more specific. Who was responsible for providing the ethical challenge, and when did they become involved? What authority and independence did they have? What evidence were they allowed to examine? What concerns did they identify, what did they recommend and what happened when those recommendations conflicted with commercial, operational or political objectives? If the organisation decided to proceed despite a material ethical concern, who signed their name to that decision and accepted responsibility for it?

That is why AI ethics may ultimately become less about publishing lists of principles and more about creating a profession capable of putting those principles into practice. I am not convinced that the future AI ethicist should have an automatic veto over every contentious decision, nor that ethical judgement can ever be reduced to another compliance checklist with a convenient green tick at the bottom. Indeed, doing either might undermine the very purpose of the role.

What I do think is increasingly difficult to defend is an AI project in which nobody has a defined responsibility to ask the uncomfortable questions, nobody has sufficient independence to pursue them when the answers are inconvenient, and nobody is required to record who accepted the consequences of proceeding anyway.

Their contribution may instead be ensuring that when the answer is not obvious, the question cannot simply be avoided—and that someone is required to ask it before the machine is switched on.