Explainable artificial intelligence

Radiopaedia • CC-by-nc-sa 3.0 • de

Explainable artificial intelligence (XAI) usually refers to narrow artificial intelligence models made with methods that enable and enhance human understanding of how the models reached outputs in each case. Many older AI models, e.g. decision trees, were inherently understandable in terms of how they produced specific results as they could be distilled into a set of clear rules. Unfortunately many state of the art models which are much better performing in terms of prediction than older models e.g. deep neural nets, have problems in terms of explainability . Ideally human users should understand not only the results generated from artificial intelligence algorithms but also salient features of how they were derived. The development of explainable artificial intelligence seeks to introduce more transparency and enable verification of conclusions from algorithms, making conclusions clearly justifiable or contestable for a user. Explainable AI is contrasted 'black-box' approaches but it should be noted that in reality some decisions of some clinicians and radiologists appear to originate in a ‘black-box’, i.e. the notoriously fickle and opaque human brain, as well.

XAI is a rapidly developing area of research that seeks to address such issues and by introducing design elements and other techniques to enable human experts to verify AI decision making.

Examples of explainable artificial intelligence techniques:

  • saliency 'heat' maps (such those as in some anomaly detection programs)

  • attention matrixes that visualize what the network is ‘looking’ at

  • weight matrixes that identify which features carry the largest impact on the algorithm

There are potential problems with employing explainable AI which depend upon the use case and specific technique involved. In some cases these techniques may increase the computational ‘weight’ of algorithms (which implies more servers and cost or slower operations ) or even increase the risk of deception about the model, as explanations may be partial or misleading e.g. certain kinds of heat maps can tell you where in an image is important, but not what features of that area made it important.

Potential benefits of employing explainable artificial intelligence:

  • human readability

  • easier justification of computational decision making

  • increased transparency and accountability

  • reducing various biases

  • avoiding discrimination