Explainability & Interpretability for Responsible AI

I put off thinking about explainability-as-a-governance-problem for an embarrassingly long time. I knew SHAP. I knew LIME. I could generate feature importance plots in my sleep. But every time someone from legal asked me "can the applicant understand why they were denied?", or a regulator asked "can you prove this model isn't discriminating?", my SHAP waterfall chart suddenly felt like bringing a calculator to a courtroom. The technical XAI tools are one thing. The obligation to explain — to real people, under real laws, with real consequences — is a completely different beast. Here is that dive.

Explainability for Responsible AI sits at the intersection of technical interpretability (Chapter 16, Section 2 covered the tools) and the legal, ethical, and social demands that surround deployed AI systems. It's the part where you stop asking "what did the model attend to?" and start asking "can a person who was harmed by this decision understand it well enough to challenge it?" The EU's GDPR codified a version of this question into law in 2018. The EU AI Act, finalized in 2024, raised the stakes dramatically.

Before we start, a heads-up. We're going to be talking about legal frameworks, documentation standards, auditing methods, and the messy human side of explanation. You don't need a law degree or prior knowledge of any of these. We'll add what we need, one piece at a time.

This isn't a short journey, but I hope you'll be glad you came.

A Loan Application and the Question It Raises

Let's start with a scenario we'll return to throughout. Priya applies for a home loan through an online lender. She fills out the form, clicks submit, and gets a rejection email within seconds. The email says: "After careful review, we are unable to approve your application at this time."

That's it. No reason. No explanation. No suggestion for what she could change.

Priya is confused. She has a steady job, decent savings, no bankruptcies. She suspects the decision was automated — there's no way a human reviewed her application in 11 seconds. And now she has a question that sounds very simple: why?

That "why" is the thread we'll follow. It turns out answering it properly requires navigating law, ethics, technical tradeoffs, documentation standards, and a fundamental tension between accuracy and understandability. Priya's question sits at the center of all of it.

The Right to Explanation — Where the Law Stands

When the European Union's General Data Protection Regulation (GDPR) went into force in 2018, it included a provision that sent a quiet shockwave through the ML world. Article 22 says, in essence: a person has the right not to be subject to a decision based solely on automated processing that produces legal effects concerning them, or that similarly significantly affects them. A loan denial qualifies.

There are exceptions — the decision can be automated if it's necessary for a contract, or if the person gives explicit consent, or if a member state's law permits it. But even when one of those exceptions applies, the organization must implement safeguards: the right to obtain human intervention, the right to express a point of view, and the right to contest the decision.

Now here's the part that generated years of academic debate. The actual text of Article 22 doesn't say the words "right to explanation." That phrase comes from Recitals 71 and 63, and from Article 15 (the right of access), which together say the data subject has a right to "meaningful information about the logic involved, as well as the significance and the envisaged consequences" of the automated processing. Whether this constitutes a legally enforceable right to an explanation or a weaker transparency obligation — legal scholars still argue about it. I'll be honest: I read five different law review articles and came away with five different interpretations.

What's not debatable is the practical reality. If you're deploying an automated decision system in the EU that affects people's lives — loan approvals, hiring decisions, insurance pricing — you need to be able to explain what happened and why. The question isn't whether you need an explanation. It's what kind, for whom, and how good it needs to be.

The EU AI Act, finalized in 2024, pushed the bar higher. It classifies AI systems by risk tier. Systems used in credit scoring, hiring, criminal justice, and healthcare are "high-risk" and face strict transparency requirements under Article 13: they must be designed so that deployers can "reasonably understand the system's functioning." Technical documentation is mandatory. Penalties for non-compliance run up to €35 million or 7% of global annual turnover. That's not a typo. Seven percent.

Think of the regulatory landscape like the building code for a house. GDPR is the foundation — it says you need walls and a roof (basic transparency). The AI Act is the detailed inspection checklist — it specifies the thickness of the walls, the fire exits, and the load-bearing requirements. And the penalties for skipping the inspection are severe enough to bankrupt you.

But here's where the law's limitations show up. It tells you that you need to explain. It tells you the explanation must be "meaningful." It doesn't tell you what a meaningful explanation actually looks like — and that's where the technical and ethical challenges really begin.

Who Needs the Explanation? Stakeholder-Specific Explanations

Back to Priya's loan denial. The lender's system does produce an explanation internally — a SHAP waterfall chart showing that her debt-to-income ratio contributed +0.34 to the rejection probability, her credit history length contributed -0.12 (a positive signal), and her employment tenure contributed +0.08. There are twelve more features in the list.

This is useful. But useful to whom?

The same decision needs to be explained to at least four very different audiences, and each one needs something entirely different. Imagine a single medical test result — the lab report means one thing to the patient, another to the GP, another to the specialist, and another to the hospital's insurance auditor. Same data, four different explanations.

The affected individual — Priya herself — needs something she can act on. She doesn't care about Shapley values or marginal contributions. She needs a sentence: "Your debt-to-income ratio of 42% exceeds our threshold. If you reduced your monthly debt payments by approximately $400, you would likely qualify." The explanation must be in plain language, actionable, and honest about what she can change versus what she can't. A counterfactual explanation works well here: "what would need to be different for the outcome to change?"

The developer — the data scientist who built the model — needs the full technical picture. SHAP values, feature interactions, partial dependence plots, the model's performance on subgroups. The developer's explanation is diagnostic: it's for finding bugs, catching bias, and debugging edge cases where the model does something unexpected.

The regulator — the data protection authority auditing the lender — needs documentation. Not a single prediction's explanation, but evidence that the system is fair in aggregate, that it was tested on relevant subgroups, that the training data was appropriate, and that the organization has a process for handling complaints. The regulator's explanation is forensic: it reconstructs how the system was built and validates that the builder followed proper procedures.

The deployer or business user — the loan officer who manages the system — needs to know when to trust the model and when to override it. They need a confidence indicator, a flag for edge cases, and enough context to make the final call when human review is triggered. Their explanation is operational: it's for making a decision about the model's decision.

I spent a long time thinking these four perspectives were a nice categorization exercise that didn't change anything in practice. I was wrong. The failure mode I see most often in production systems is building one explanation and serving it to everyone. The developer gets the SHAP chart, and when the regulator asks for documentation, someone screenshots the SHAP chart and puts it in a PDF. When the applicant asks why they were denied, someone paraphrases the SHAP chart in slightly simpler language. None of these audiences actually gets what they need.

The explanation isn't one thing. It's four different things, built for four different purposes, with four different levels of technical detail. And building all four is genuinely expensive, which is why most organizations only build one.

The Fidelity-Comprehensibility Tradeoff

Here's a tension that sits underneath all of this, and once you see it, you can't unsee it.

Fidelity is how accurately an explanation reflects what the model actually does. A perfectly faithful explanation is the model itself — every weight, every nonlinear interaction, every edge case. Comprehensibility is how well a human can understand the explanation. A perfectly comprehensible explanation might be: "you were rejected because your debt is too high."

The problem: these two properties pull in opposite directions. And not a little bit — they pull hard.

Let's go back to Priya's loan model. The full model considers 47 features with nonlinear interactions between them. Her debt-to-income ratio matters, but its effect depends on her employment sector, her savings trajectory over the past 6 months, and whether she lives in a postal code with rising or declining property values. The faithful explanation of her rejection would involve all of this. No human — not Priya, not the developer, not the regulator — can hold a 47-dimensional interaction space in their head.

So we simplify. We say "debt-to-income ratio was the most important factor." That's comprehensible. But it's also a lie — or at minimum, a lossy compression. The model didn't make the decision based on debt-to-income alone. It made the decision based on a complex function of 47 variables, and we're projecting that onto a single axis. The simplification can hide exactly the patterns we care about: maybe the model is penalizing her postal code in a way that correlates with race. That information gets compressed away when we simplify.

Think of it like a map. A subway map of London is wonderfully comprehensible — you can plan your route in seconds. But it's geometrically unfaithful. The distances between stations are wrong, the curves are straightened, entire neighborhoods disappear. A satellite photo of London is perfectly faithful — every street, every building. But try planning your commute on it. The subway map and the satellite photo are both "explanations" of London's geography. Neither is wrong. They serve different purposes at different fidelity levels.

The research community has been attacking this tradeoff from several angles. Hybrid explanations layer a simple summary on top of a more detailed technical breakdown — the subway map with an option to zoom into the satellite photo. User-adaptive explanations adjust their complexity based on who's asking. And fidelity metrics try to quantify exactly how much accuracy you're losing when you simplify, so at least you know the cost of comprehensibility.

But I want to be honest: nobody has solved this. Every explanation you've ever seen of a complex model is a simplification, and every simplification hides something. The question is never "is this explanation complete?" — it's "is what it hides acceptable for this audience and this purpose?"

Model Documentation — Model Cards and Datasheets

Individual prediction explanations are important, but they're not enough. A regulator doesn't want to audit one decision at a time — they want to understand the system. A developer inheriting a colleague's model doesn't need to see one SHAP plot — they need to know what the model was trained on, what it's supposed to do, where it fails, and what biases were tested for.

This is where documentation standards come in. And I'll confess: I thought model documentation was bureaucratic busywork until I inherited a production model with no documentation and spent three weeks figuring out what the training data even was.

A model card, proposed by Mitchell et al. in 2019, is the closest thing ML has to a nutrition label. Think about what a nutrition label does: it doesn't tell you whether the food is "good" or "bad" — it gives you enough information to make that judgment yourself based on your needs. A model card does the same thing for a trained model. The key sections are:

Model Details — who built it, when, what architecture, what version. This sounds trivial until you discover that a production model has been silently retrained three times and nobody recorded which version is currently serving.

Intended Use — what the model is designed for and what it's explicitly not designed for. For Priya's loan model: "intended for preliminary credit screening of individual consumer loan applications in the UK market. Not intended for business lending, mortgage underwriting, or applications outside the UK."

Performance Metrics — not a single accuracy number, but performance broken down by relevant subgroups. How does the model perform on applicants over 50? On applicants from rural areas? On applicants with thin credit files? These disaggregated metrics are where hidden biases surface.

Training Data — what data was used, how it was collected, what time period it covers, what preprocessing was applied. For Priya's model, this might reveal that the training data contains no applicants from Northern Ireland, which means the model's predictions for that population are extrapolations, not interpolations.

Ethical Considerations — known risks, potential for misuse, tested-for biases. This is the section most organizations leave blank, and it's the one regulators read first.

Caveats and Recommendations — limitations, failure modes, things the next person needs to know. "Model performance degrades significantly for applicants with less than 2 years of credit history. Consider manual review for these cases."

Datasheets for datasets, proposed by Gebru et al. in 2021, apply the same philosophy to data. They ask questions like: Why was this dataset created? Who funded it? What's in it? How was consent obtained? What are its known limitations? The datasheet for the training data behind Priya's loan model should disclose that the data comes from a single lender's historical decisions — decisions that were themselves potentially biased.

The nutrition label analogy extends further. A nutrition label doesn't prevent you from eating junk food — it just makes it harder to do so accidentally. Model cards and datasheets don't prevent irresponsible AI — they make it harder to deploy irresponsible AI while claiming you didn't know.

IBM extended this idea with FactSheets, which cover the full AI service lifecycle including deployment context and operational monitoring. And for NLP specifically, Data Statements (Bender and Friedman, 2018) capture sociolinguistic information — who the speakers are, what dialects are represented, what demographics were included — that's critical for understanding whose language a model actually understands.

Algorithmic Auditing — Checking Your Own Work

Documentation tells you what a system is supposed to do. Auditing tells you what it actually does. The difference matters more than I initially expected.

An algorithmic audit is a systematic examination of an AI system's behavior, inputs, and outputs to assess whether it meets specified standards for fairness, accuracy, safety, or compliance. Think of it like a financial audit: you're not taking the company's word for how much money they have. You're independently verifying the books.

Let's walk through what an audit of Priya's loan model might look like.

The first layer is an internal audit — the team that built the model tests it themselves. They run the model on held-out data, disaggregate performance by protected attributes (race, gender, age), check for disparate impact, and verify that the features don't include prohibited variables or close proxies. This is necessary but insufficient, for the same reason that you can't grade your own homework. You're too close to the work. You've made assumptions you don't even know you've made.

The second layer is an external audit — an independent third party examines the system. Organizations like ForHumanity, BSI, and TüV SÜD now offer AI audit services. An external auditor might run their own test cases, examine the training data pipeline, interview the development team, and assess the contestability mechanisms. They bring fresh eyes and no incentive to find the system acceptable.

The third layer is a regulatory audit — a government authority examines the system for compliance with specific laws. Under the EU AI Act, high-risk AI systems are subject to conformity assessments before they can be placed on the market. The organization must maintain technical documentation, logging, and incident reports that auditors can access.

The NIST AI Risk Management Framework (AI RMF), released in 2023, provides a structured approach to this: identify risks, measure them, mitigate them, document everything. It's not prescriptive about specific techniques — it's a meta-framework that tells you what questions to ask and what evidence to produce. ISO/IEC 42001:2023 goes further, providing a certifiable management system standard for organizations using AI.

Algorithmic Impact Assessments (AIAs) flip the audit timeline. Instead of auditing after deployment, an AIA evaluates the system's likely societal impacts before deployment. What populations will be affected? What are the potential harms? What mitigation measures are in place? Canada's government has required AIAs for federal automated decision systems since 2019 — it's one of the earliest mandated pre-deployment assessment frameworks in the world.

Here's what I find compelling about this layered approach: no single audit catches everything. Internal auditors know the system deeply but have blind spots. External auditors bring independence but less context. Regulatory auditors enforce compliance but may not catch subtle technical issues. Each layer compensates for the others' weaknesses. It's defense in depth, applied to governance.

Contestability — The Right to Fight Back

We've talked about explaining decisions and auditing systems. But there's a piece that often gets overlooked, and it might be the most important one: what happens when the person affected by the decision thinks it's wrong?

Contestability is the principle that individuals should be able to challenge automated decisions that affect them. Not politely request a reconsideration — actually challenge, with a meaningful process and a real possibility that the decision gets reversed.

GDPR Article 22 embeds this directly: when automated decisions with significant effects are made, the person has the right to obtain human intervention, express their point of view, and contest the decision. The EU AI Act reinforces it for high-risk systems. UNESCO's 2021 Recommendation on the Ethics of AI calls for contestability mechanisms globally.

What does good contestability look like in practice? Let's build it out for Priya's loan case.

First, notification. Priya must be told that the decision was automated. Not buried in page 47 of the terms of service — proactively, at the time the decision is communicated. "This decision was made with the assistance of an automated system."

Second, explanation. She needs enough information to understand the basis for the decision. Not "your application didn't meet our criteria" but "your debt-to-income ratio of 42% and your credit history length of 18 months were the primary factors."

Third, a clear appeals process. How does she contest? Who does she contact? What's the timeline? If the appeals process is a mailto: link to a general inbox that nobody checks, it's not a real process.

Fourth, meaningful human review. When Priya appeals, an actual human reviews her case substantively — not a human who rubber-stamps the model's output. This is harder than it sounds. If the loan officer always defers to the model because "the model knows better," the human review is theater.

Fifth, feedback loops. Successful contestations should feed back into the system. If Priya's appeal reveals that the model is systematically undervaluing applicants with non-traditional income sources, that's a training data problem that should be fixed.

I'll be honest — I've seen very few organizations implement all five of these well. Most get the notification right, provide a mediocre explanation, and have an appeals process that's more symbolic than functional. Meaningful human review is expensive. Feedback loops require engineering investment. Contestability, done properly, is a product feature — not a checkbox.

Transparency Reports — Showing Your Work Publicly

Model cards document a single model. Audits verify a single system at a point in time. Transparency reports are something broader: they're public-facing documents that describe an organization's AI practices, decisions, and outcomes in aggregate.

The idea isn't new — tech companies have published transparency reports about government data requests for years. Applying the concept to AI systems means disclosing things like: how many automated decisions were made this quarter, what percentage were contested, how many contestations were upheld, what changes were made to the system as a result, and what the aggregate performance looks like across demographic groups.

A good transparency report for Priya's lender might include: the total number of applications processed automatically (say, 240,000), the approval rate broken down by age, gender, and region, the number of appeals (1,800), the number of appeals that resulted in a reversed decision (340), and any model retraining or updates that occurred. It would also disclose what external audits were conducted and what they found.

Nobody is going to pretend that organizations publish these reports joyfully. There's a real competitive concern: disclosing approval rates by demographic group might reveal patterns that invite regulatory scrutiny or public criticism, even if the patterns reflect legitimate risk differences. But the alternative — operating opaque systems that affect millions of people with no public accountability — is increasingly untenable. The EU AI Act requires that deployers of high-risk AI systems maintain logs and cooperate with regulators. Some jurisdictions are moving toward mandatory public transparency for automated decision systems used by government agencies.

The building code analogy from earlier is useful again. A transparency report is like a building's certificate of occupancy posted in the lobby. It doesn't prove the building is safe — it proves that someone checked, and tells you who to call if you disagree.

Explanation in High-Stakes Domains

Everything we've built up so far — the right to explanation, stakeholder-specific communication, documentation, auditing, contestability — gets tested hardest in domains where the consequences of a bad decision are irreversible or severe. Three domains stand out, and each one illuminates a different failure mode.

Criminal Justice

The case that made algorithmic explainability a public conversation was COMPAS — the Correctional Offender Management Profiling for Alternative Sanctions system, used across the United States to predict the likelihood that a defendant will reoffend. Judges use COMPAS scores to inform bail, sentencing, and parole decisions.

In 2016, ProPublica published an investigation showing that COMPAS was significantly more likely to falsely label Black defendants as high-risk compared to white defendants with similar profiles. The false positive rate for Black defendants was nearly twice that for white defendants.

But the explainability failure was arguably worse than the bias itself. COMPAS is proprietary. Defendants couldn't inspect the model. Defense attorneys couldn't challenge its logic. In State v. Loomis (2016), a Wisconsin defendant argued that the use of COMPAS in his sentencing violated his due process rights because the methodology was a trade secret. The Wisconsin Supreme Court ruled that COMPAS could be used — but acknowledged that its proprietary nature was problematic and that sentencing courts should exercise caution.

Think about what that means. A person's freedom is being influenced by a score that nobody outside the company can verify. The company says "trust us." And the court said "proceed with caution." That gap between "trust us" and "here's how it works, check for yourself" is the explainability gap — and in criminal justice, it's the gap between a functional justice system and algorithmic sentencing theater.

Healthcare

In healthcare, the explainability challenge takes a different shape. An AI system that recommends a treatment or flags a diagnosis needs to produce an explanation that a clinician can integrate into their existing clinical reasoning — not replace it.

A deep learning model that reads chest X-rays and says "pneumonia, 0.87 confidence" is useful. The same model that says "pneumonia, 0.87 confidence — here's a GradCAM heatmap highlighting the lower left lobe" is more useful, because the clinician can check whether the model is looking at the right part of the image. But even the heatmap has limits: it shows where the model looked, not why it concluded pneumonia rather than, say, pleural effusion.

The deeper issue in healthcare is that explanation serves a dual function. It's not enough for the doctor to understand the model — the patient needs to understand the decision too. A patient told "our AI system recommends this treatment" is in a fundamentally different position than a patient told "based on your imaging results and similar cases, this treatment has a 73% success rate — here's what the alternative options look like." The second explanation enables informed consent. The first one doesn't.

I'm still developing my intuition for where AI explanation and clinical explanation should merge versus remain separate. But the direction is clear: healthcare AI explanations need to serve both the clinician's diagnostic workflow and the patient's autonomy. One explanation can't do both.

Finance

Finance has the longest history of regulated explainability, which means it's the domain where the gap between legal requirements and actual practice is most visible. In the United States, the Equal Credit Opportunity Act (ECOA) requires that lenders provide specific reasons for adverse credit decisions — not "your score was too low" but specific factors like "ratio of revolving balances to credit limits is too high." The Fair Credit Reporting Act (FCRA) gives consumers the right to dispute inaccurate information that affects credit decisions.

These laws predate ML by decades. They were written for rule-based credit scoring systems where the "reasons" were literal rules. When a gradient-boosted ensemble replaces the rule-based system, the legal requirement doesn't change — you still need to provide specific reasons — but the technical challenge of extracting those reasons from a nonlinear model becomes non-trivial.

This is where Priya's story connects back to technical XAI. The lender uses SHAP values to identify the top contributing features and translates them into the legally-required "reason codes." But as we discussed earlier, SHAP values are approximations that may not perfectly reflect the model's actual logic. If Priya disputes the decision and an investigation reveals that the stated reasons don't match the model's actual behavior, the lender has a legal problem — not a technical one.

What ties all three domains together is that explainability isn't a nice-to-have that gets bolted on at the end. It's a structural requirement that shapes how the system is designed, deployed, documented, and contested from the start.

The Rashomon Problem — When Explanations Aren't Unique

Here's something that genuinely unsettled me when I first encountered it, and I still think about it.

For most real-world datasets, there isn't one "best" model. There's a set of models — sometimes a very large set — that all achieve essentially the same accuracy. Cynthia Rudin and others call this the Rashomon set, named after the Akira Kurosawa film where different witnesses give contradictory accounts of the same event.

Two models in the Rashomon set might both predict Priya's loan rejection correctly. But one model might say the primary reason was her debt-to-income ratio, and the other might say it was her employment tenure. Both models have the same accuracy. Both explanations are faithful — to their respective models. And they contradict each other.

This is deeply uncomfortable for anyone who wants to use explanations as the basis for contestability or regulation. If Priya receives an explanation that says "debt-to-income ratio was the main factor," she might take action to reduce her debt. But if a different, equally valid model would have given a different explanation, was the first explanation real? Or was it an artifact of an arbitrary model selection?

The Rashomon effect has a constructive implication too. If the Rashomon set contains models that are both accurate and interpretable, we don't need to use a black box at all. We can search the set for a model that's inherently understandable. Rudin has argued persuasively that for high-stakes decisions, the existence of a large Rashomon set is actually a reason to require interpretable models — because with so many good models to choose from, the excuse that "we need a black box for accuracy" is much weaker.

My favorite thing about the Rashomon problem is that it forces honesty. It means that any single explanation of a complex model's decision is one of many possible explanations, and anyone who presents one explanation as "the" explanation is, at best, being incomplete. At worst, they're choosing the explanation that makes them look best.

Putting It All Together

If you're still with me, thank you. I hope it was worth it.

We started with Priya's simple question — "why was I denied?" — and followed it into a surprisingly deep thicket of law (GDPR Article 22, the EU AI Act), stakeholder-specific communication (four audiences, four different explanations), the fundamental tradeoff between faithful and understandable explanations, documentation standards (model cards, datasheets, FactSheets), auditing at three layers (internal, external, regulatory), contestability as a system requirement, transparency reports as public accountability, domain-specific challenges (criminal justice, healthcare, finance), and the Rashomon problem that calls into question whether explanations are even unique.

The thread through all of it is that explainability for responsible AI isn't a technical feature — it's a sociotechnical system. The SHAP values are one component. The documentation is another. The appeals process is another. The transparency report is another. Each one is necessary. None is sufficient alone.

My hope is that the next time someone from legal or compliance asks "can we explain this model's decisions?", instead of reaching for a SHAP plot and calling it done, you'll think about who needs the explanation, what they need to do with it, and whether the system is designed to support that — from the documentation to the appeals process to the public transparency. That's a much harder question. But it's the right one.

Resources and Credits