The AI community has introduced various concepts and tools to interpret AI model outcomes, including feature importance, partial dependence plots, and sub-population analysis. The Explainable AI (XAI) tools are crucial in building trust among end-users and regulators, identifying and mitigating bias, and improving overall model performance. They are built to answer the main question of all users: “Why did the model make a specific prediction for an instance or a group of instances?”
While the XAI tools are invaluable in identifying bias and building trust, they are highly susceptible to misuse.
For instance, most feature importance methods assume that features are independent. As a result, including highly correlated features in the analysis can lead to unreliable outcomes. Moreover, different approaches for calculating the global importance of features, such as using the “mean…