The A-level crisis
August 2020. In the UK, thousands of students opened their A-level results to find their grades downgraded by an algorithm. No clear reason was given. Teachers and families could not see the logic behind the scores. Within days, complaints turned to protests. Public trust collapsed and the system was abandoned.
The lesson was not simply that the model had flaws. It was that people could not understand or challenge the decisions that shaped their futures. A system can be accurate to its metrics yet meaningless to those it governs. When people cannot see the reasons, trust erodes fast. The risk is not only technical failure, but the invisibility of decisions that matter.
A system can be technically flawless yet socially unreadable. The risk is not failure, but the invisibility.
The black box problem
AI models, especially those using machine learning, often resist human interpretation. They can detect patterns across millions of data points and generate outputs that outperform humans in accuracy. But the logic behind these outcomes is rarely clear.
In healthcare, an AI tool might flag a tumour as malignant with 95 per cent confidence, yet offer no explanation of why. In finance, a model might decline a loan application because of “risk”, without showing which factors mattered most. In public services, such opacity creates an empathy gap. It undermines fairness, reduces accountability and leaves users unable to recover from errors.
Research highlights this tension. Tim Miller argues that explanations are a social act, not just a technical one. People want to know “why” in relation to their goals and context. Without this, services risk amplifying distrust rather than solving it.2
From transparency to explainability
One way to see this is to distinguish between different kinds of understanding. In AI, three terms often overlap:
Transparency
Making the workings visible, for example listing the variables a model uses.
Interpretability
Ensuring behaviour can be understood in human terms, such as linking “income stability” to loan risk.
Explainability
Framing outputs in ways that make sense to the user, such as “Your application was declined because recent income changes make it difficult to assess long-term affordability.”
Each level has value. But explainability is where design practice makes the difference. It shifts the focus from how the model works to how the user experiences its decisions.
A user-centred lens on explainable AI
To make this concrete, the US National Institute of Standards and Technology (NIST) set out four principles for explainable AI in 2020. These are explanation, meaning, trust and fairness. Together they provide a user-centred way of checking whether system performance also makes sense to the people who rely on it.3
These principles move beyond system-centric benchmarks like speed and error rates to focus on what makes services legitimate, contestable and human.
A practical blueprint
How should public service teams approach explainable AI? Evidence from HCI and human factors research suggests three steps:
Start with the questions people will ask. Miller shows that users frame explanations in terms of their goals (“Why me?”, “What next?”). Designing for these questions anchors the system in lived experience.2
Lipton highlights counterfactuals (“If X had been different, Y would have changed”) and feature attribution as two of the most effective methods for aligning model outputs with human reasoning.1
Test for recoverability. Sarter, Woods and Billings argue that recovery paths are critical to trust in automation. A system that cannot be challenged cannot be relied on.6
These steps do not remove complexity. But they anchor automation in user-centred practice.
Cautionary tales
The risks of opacity are real. In US healthcare, a widely used algorithm for predicting patient risk was found to underestimate the needs of some patient groups. Because it used past healthcare spending as a proxy for future need, it reproduced existing racial disparities. The system was technically accurate to its chosen metric but socially flawed in practice.5
Trust collapses fastest when people cannot see or contest a decision.
By contrast, radiology tools that highlight areas of a scan for clinician review show how explainability can build confidence. The AI does not replace judgement but supports it, making its reasoning visible and actionable.
What success really means
The A-levels crisis in the UK showed what happens when decisions cannot be explained. The algorithm was abandoned not because it failed technically, but because people felt shut out. Trust collapsed in days.
Accuracy on its own will never be enough. Public services succeed when people can see why a decision was made, trust the reasoning, and know what to do next. If efficiency comes at the expense of fairness, it is not progress. If models are optimised but people cannot trust them, legitimacy is lost.
Our role is to keep both sides in balance. The future of automation in government will not be decided by accuracy scores, but by whether citizens can understand and live with the outcomes. Designing for explainability is how we make that possible.
Sources and further reading
- Lipton, Z. C. (2018). The mythos of model interpretability. Communications of the ACM, 61(10), 36–43.
- Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38.
- NIST (2020). Four Principles of Explainable Artificial Intelligence. National Institute of Standards and Technology.
- Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
- Sarter, N. B., Woods, D. D., & Billings, C. E. (1997). Automation surprises. In G. Salvendy (Ed.), Handbook of Human Factors and Ergonomics. Wiley.