Explainability: Why Accurate Prediction Is Not Yet an Explanation

A system can repeatedly predict the right outcome without showing why it succeeds. Umbrella sales may predict rain in a district, but they do not cause the rain. Likewise, a model can achieve high accuracy by exploiting a stable correlation without revealing which factors genuinely matter.

Explainability requires an output to come with understandable reasons and to state the conditions under which those reasons hold. It differs from transparency: seeing every line of code and every parameter may still leave a particular decision obscure. It also differs from causal explanation, because an account can faithfully describe how a model used information without claiming to have discovered causes in the world.

The distinction matters most in high-stakes decisions. If a lending model rejects an applicant, we want more than its usual accuracy. We need to know what the decision depended on, which relevant change might alter the result, and when the model has crossed its knowledge limits. Explainability is therefore not a pleasant story pasted onto complexity. It is a checkable chain of reasons. Accuracy asks, “Is it often right?” Explanation asks, “Why this judgement, and when should it not be trusted?”

https://www.nist.gov/publications/four-principles-explainable-artificial-intelligence
https://www.nature.com/articles/s42256-019-0048-x