What Counts as an “Explanation” in AI?

 

Philosophy of Explainability (Part 1)

What Counts as an “Explanation” in AI?

Introduction: The Human Demand for Explanations

We humans are explanation-seeking creatures. From children asking “Why is the sky blue?” to scientists probing the origins of the universe, we crave reasons. An explanation is not just extra information—it is the bridge between cause and understanding. But when it comes to AI (Artificial Intelligence), particularly modern LLMs (Large Language Models) and neural networks, this demand runs into trouble: what does it even mean to explain the output of a system whose inner workings defy human comprehension?

Philosophical Roots: What Is an Explanation?

In the philosophy of science, explanations are categorized into a few major types:

These categories remind us that explanations are not one-size-fits-all. They are tools designed to match the questioner’s need.

Explanations in AI: Whose Explanation?

AI complicates this further. Consider a loan approval system trained on millions of data points. If you’re denied a loan, what explanation do you deserve?

  • To a developer, an explanation might mean: “The model assigned you a creditworthiness score below 0.55 because of how features like income-to-debt ratio interacted in the hidden layers.”

  • To a regulator, it might mean: “The system’s training data produced disparate impact by gender, violating fairness guidelines.”

  • To a customer, it might mean: “You were denied because your income-to-debt ratio is too high compared to similar applicants.”

All three are explanations—but they differ in audience and purpose.

This highlights a crucial point: in AI, an explanation is audience-relative. What counts as an explanation depends not just on the system, but on who is asking and why.

The Problem of Post Hoc Explanations

Many AI systems—particularly deep learning models—are “black boxes.” We can measure inputs and outputs, but the path in between is too complex to map step by step. To cope, researchers use post hoc explanations like:

  • Saliency maps: Heatmaps that show which features of an image influenced a classification.

  • LIME (Local Interpretable Model-agnostic Explanations): A method that perturbs input data and observes output changes to approximate “why” a model made a prediction.

  • SHAP values (SHapley Additive exPlanations): Borrowed from cooperative game theory, assigning each feature a “contribution” to the outcome.

But these are approximations. They don’t reveal the true inner logic—they provide a simplified, human-readable story.

This mirrors cognitive science research on humans: when asked “Why did you do that?”, people often construct explanations after the fact that are plausible but not necessarily causal. AI, in a strange way, is just holding up a mirror.

Case Study: COMPAS Recidivism Predictions

The COMPAS algorithm, used in US courts to predict re-offense risk, offers a sobering example. Investigations revealed racial bias: Black defendants were more likely to be labeled “high risk” even when they did not re-offend.

Here the demand for explanation was both technical and moral: How did COMPAS make this judgment? The company claimed “trade secret” protection, so full transparency was unavailable. Instead, journalists and researchers reverse-engineered explanations using input-output analysis.

This case shows that explanation is not just about technical clarity—it is about social trust. Without accessible explanations, public legitimacy collapses.

Critical Thinking Prompts

  • When is an explanation “good enough”?

  • Should AI explanations aim for truth (the real causal story) or usefulness (a simplified model humans can act on)?

  • How should we handle cases where different audiences (developers, policymakers, end-users) require fundamentally different explanations?

Conclusion: Explanation as Dialogue

Philosophically, the lesson is this: explanations are not static facts waiting to be uncovered. They are dialogues between systems, humans, and contexts. For AI, this means:

  1. We need pluralistic explanations tailored to audience needs.

  2. We must accept that some explanations will be approximations, not exact causal maps.

  3. The ultimate test of explanation is not whether it is complete, but whether it supports human understanding and decision-making.

In short: What counts as an explanation in AI? The answer depends on who you are, what you need, and how much complexity you can handle.

Comments

Post a Comment

Popular posts from this blog

Interpretability vs. Explainability: Why the Distinction Matters

Image
Interpretability vs. Explainability: Why the Distinction Matters When we talk about XAI (Explainable Artificial Intelligence) , two terms keep popping up: interpretability and explainability . They sound interchangeable, but if we treat them as synonyms, we miss an important distinction. Let’s unpack these terms carefully, because how we define them changes how we design, use, and regulate AI. Step One: Definitions (Clarity First) Interpretability means the degree to which a human can understand how a model works internally. Think of it as model transparency . Example: A decision tree is interpretable because we can follow each “if–then” branch and see exactly how the output was reached. Explainability means the degree to which a model’s output can be made understandable to a human. Think of it as output justification . Example: A deep neural network is not inherently interpretable, but we can use tools like LIME (Local Interpretable Model-Agnostic Explanations) ...
Read more

Healthcare AI: The Role of Explainability in Diagnostics

Healthcare AI: The Role of Explainability in Diagnostics — Industry Case Studies in XAI Industry Case Studies in XAI Glossary Audiences Case Studies Comparison Checklist Pause & Probe Blogger Series • Critical Thinking for Collaboration Healthcare AI: The Role of Explainability in Diagnostics XAI (Explainable Artificial Intelligence) refers to techniques that make AI models transparent , interpretable , and justifiable to humans. In healthcare, explainability is not a cosmetic add‑on. It is a clinical safety feature, an ethical commitment, and a legal anchor. When an algorithm influences a diagnosis, clinicians, patients, and regulators each need an explanation— but not the same one . In this article, we map the diagnostic AI landscape, distinguish the informational needs of different audiences, and walk through real‑world patterns for medical imaging , clinical decision sup...
Read more

“How FinTech Firms Use XAI to Build Trust”

  Industry Case Studies in Explainable AI (XAI) Industry Case Studies in XAI FinTech Healthcare Self‑Driving Cars Security & Surveillance Blogger Series • Critical Thinking for Collaboration Explainable AI (XAI): Four Industry Case Studies with Comparisons, Diagrams, and Critical Questions XAI (Explainable Artificial Intelligence) means techniques that make AI decisions transparent , interpretable , and justifiable to humans. Throughout, we refresh key terms and avoid jargon by spelling it out — e.g., LLM (Large Language Model) , ROC (Receiver Operating Characteristic) , AUC (Area Under the Curve) . Each article ends with Pause & Probe questions to train critical thinking. How FinTech Firms Use XAI to Build Trust FinTech (Financial Technology) products trade in one scarce commodity: trust . If an app manages your savings, approves your mortgage, or freeze...
Read more