When Shapley Values Break: A Information to Strong Mannequin Explainability

Explainability in AI is crucial for gaining belief in mannequin predictions and is extremely essential for enhancing mannequin robustness. Good explainability typically acts as a debugging software, revealing flaws within the mannequin coaching course of. Whereas Shapley Values have grow to be the business normal for this process, we should ask: Do they all the time work? And critically, the place do they fail?

To know the place Shapley values fail, the perfect strategy is to regulate the bottom reality. We’ll begin with a easy linear mannequin, after which systematically break down the reason. By observing how Shapley values react to those managed adjustments, we are able to exactly determine precisely the place they yield deceptive outcomes and repair them.

The Toy Mannequin

We’ll begin with a mannequin with 100 uniform random variables.

import numpy as np
from sklearn.linear_model import LinearRegression
import shap

def get_shapley_values_linear_independent_variables(
    weights: np.ndarray, information: np.ndarray
) -> np.ndarray:
    return weights * information

# High examine the theoretical outcomes with shap package deal
def get_shap(weights: np.ndarray, information: np.ndarray):
    mannequin = LinearRegression()
    mannequin.coef_ = weights  # Inject your weights
    mannequin.intercept_ = 0
    background = np.zeros((1, weights.form[0]))
    explainer = shap.LinearExplainer(mannequin, background) # Assumes impartial between all options
    outcomes = explainer.shap_values(information) 
    return outcomes

DIM_SPACE = 100

np.random.seed(42)
# Generate random weights and information
weights = np.random.rand(DIM_SPACE)
information = np.random.rand(1, DIM_SPACE)

# Set particular values to check our instinct
# Characteristic 0: Excessive weight (10), Characteristic 1: Zero weight
weights[0] = 10
weights[1] = 0
# Set maximal worth for the primary two options
information[0, 0:2] = 1

shap_res = get_shapley_values_linear_independent_variables(weights, information)
shap_res_pacakge = get_shap(weights, information)
idx_max = shap_res.argmax()
idx_min = shap_res.argmin()

print(
    f"Anticipated: idx_max 0, idx_min 1nActual: idx_max {idx_max},  idx_min: {idx_min}"
)

print(abs(shap_res_pacakge - shap_res).max()) # No distinction

On this easy instance, the place all variables are impartial, the calculation simplifies dramatically.

Recall that the Shapley components relies on the marginal contribution of every characteristic, the distinction within the mannequin’s output when a variable is added to a coalition of identified options versus when it’s absent.

[ V(S∪{i}) – V(S)
]

For the reason that variables are impartial, the precise mixture of pre-selected options (S) doesn’t affect the contribution of characteristic i. The impact of pre-selected and non-selected options cancel one another out throughout the subtraction, having no influence on the affect of characteristic i. Thus, the calculation reduces to measuring the marginal impact of characteristic i immediately on the mannequin output:

[ W_i · X_i ]

The result’s each intuitive and works as anticipated. As a result of there isn’t any interference from different options, the contribution relies upon solely on the characteristic’s weight and its present worth. Consequently, the characteristic with the biggest mixture of weight and worth is probably the most contributing characteristic. In our case, characteristic index 0 has a weight of 10 and a price of 1.

Let’s Break Issues

Now, we are going to introduce dependencies to see the place Shapley values begin to fail.

On this state of affairs, we are going to artificially induce good correlation by duplicating probably the most influential characteristic (index 0) 100 instances. This leads to a brand new mannequin with 200 options, the place 100 options are similar copies of our unique high contributor and impartial of the remainder of the 99 options. To finish the setup, we assign a zero weight to all these added duplicate options. This ensures the mannequin’s predictions stay unchanged. We’re solely altering the construction of the enter information, not the output. Whereas this setup appears excessive, it mirrors a typical real-world state of affairs: taking a identified essential sign and creating a number of derived options (equivalent to rolling averages, lags, or mathematical transformations) to higher seize its data.

Nonetheless, as a result of the unique Characteristic 0 and its new copies are completely dependent, the Shapley calculation adjustments.

Based mostly on the Symmetry Axiom: if two options contribute equally to the mannequin (on this case, by carrying the identical data), they need to obtain equal credit score.

Intuitively, understanding the worth of anybody clone reveals the complete data of the group. Consequently, the large contribution we beforehand noticed for the only characteristic is now cut up equally throughout it and its 100 clones. The “sign” will get diluted, making the first driver of the mannequin seem a lot much less essential than it truly is.
Right here is the corresponding code:

import numpy as np
from sklearn.linear_model import LinearRegression
import shap

def get_shapley_values_linear_correlated(
    weights: np.ndarray, information: np.ndarray
) -> np.ndarray:
    res = weights * information
    duplicated_indices = np.array(
        [0] + listing(vary(information.form[1] - DUPLICATE_FACTOR, information.form[1]))
    )
    # we are going to sum these contributions and cut up contribution amongst them
    full_contrib = np.sum(res[:, duplicated_indices], axis=1)
    duplicate_feature_factor = np.ones(information.form[1])
    duplicate_feature_factor[duplicated_indices] = 1 / (DUPLICATE_FACTOR + 1)
    full_contrib = np.tile(full_contrib, (DUPLICATE_FACTOR+1, 1)).T
    res[:, duplicated_indices] = full_contrib
    res *= duplicate_feature_factor
    return res

def get_shap(weights: np.ndarray, information: np.ndarray):
    mannequin = LinearRegression()
    mannequin.coef_ = weights  # Inject your weights
    mannequin.intercept_ = 0
    explainer = shap.LinearExplainer(mannequin, information, feature_perturbation="correlation_dependent")    
    outcomes = explainer.shap_values(information)
    return outcomes

DIM_SPACE = 100
DUPLICATE_FACTOR = 100

np.random.seed(42)
weights = np.random.rand(DIM_SPACE)
weights[0] = 10
weights[1] = 0
information = np.random.rand(10000, DIM_SPACE)
information[0, 0:2] = 1

# Duplicate copy of characteristic 0, 100 instances:
dup_data = np.tile(information[:, 0], (DUPLICATE_FACTOR, 1)).T
information = np.concatenate((information, dup_data), axis=1)
# We'll put zero weight for all these added options:
weights = np.concatenate((weights, np.tile(0, (DUPLICATE_FACTOR))))


shap_res = get_shapley_values_linear_correlated(weights, information)

shap_res = shap_res[0, :] # Take First document to check outcomes
idx_max = shap_res.argmax()
idx_min = shap_res.argmin()

print(f"Anticipated: idx_max 0, idx_min 1nActual: idx_max {idx_max},  idx_min: {idx_min}")

That is clearly not what we meant and fails to offer rationalization to mannequin habits. Ideally, we wish the reason to replicate the bottom reality: Characteristic 0 is the first driver (with a weight of 10), whereas the duplicated options (indices 101–200) are merely redundant copies with zero weight. As an alternative of diluting the sign throughout all copies, we might clearly choose an attribution that highlights the true supply of the sign.

Be aware: If you happen to run this utilizing Python shap package deal, you may discover the outcomes are related however not similar to our guide calculation. It’s because calculating Shapley values is computationally infeasible. Due to this fact libraries like shap depend on approximation strategies which barely introduce variance.

Can We Repair This?

Since correlation and dependencies between options are extraordinarily widespread, we can not ignore this challenge.

On the one hand, Shapley values do account for these dependencies. A characteristic with a coefficient of 0 in a linear mannequin and no direct impact on the output receives a non-zero contribution as a result of it incorporates data shared with different options. Nonetheless, this habits, pushed by the Symmetry Axiom, is just not all the time what we wish for sensible explainability. Whereas “pretty” splitting the credit score amongst correlated options is mathematically sound, it typically hides the true drivers of the mannequin.

A number of strategies can deal with this, and we are going to discover them.

Grouping Options

This strategy is especially vital for high-dimensional characteristic house fashions, the place characteristic correlation is inevitable. In these settings, trying to attribute particular contributions to each single variable is usually noisy and computationally unstable. As an alternative, we are able to mixture related options that symbolize the identical idea right into a single group. A useful analogy is from picture classification: if we need to clarify why a mannequin predicts “cat” as an alternative of a “canine”, analyzing particular person pixels is just not significant. Nonetheless, if we group pixels into “patches” (e.g., ears, tail), the reason turns into instantly interpretable. By making use of this similar logic to tabular information, we are able to calculate the contribution of the group slightly than splitting it arbitrarily amongst its parts.

This may be achieved in two methods: by merely summing the Shapley values inside every group or by immediately calculating the group’s contribution. Within the direct technique, we deal with the group as a single entity. As an alternative of toggling particular person options, we deal with the presence and absence of the group as simultaneous presence or absence of all options inside it. This reduces the dimensionality of the issue, making the estimation quicker, extra correct, and extra secure.

The Winner Takes It All

Whereas grouping is efficient, it has limitations. It requires defining the teams beforehand and sometimes ignores correlations between these teams.

This results in “rationalization redundancy”. Returning to our instance, if the 101 cloned options should not pre-grouped, the output will repeat these 101 options with the identical contribution 101 instances. That is overwhelming, repetitive, and functionally ineffective. Efficient explainability ought to cut back the redundancy and present one thing new to the consumer every time.

To realize this, we are able to create a grasping iterative course of. As an alternative of calculating all values directly, we are able to choose options step-by-step:

1. Choose the “Winner”: Determine the only characteristic (or group) with the very best particular person contribution
2. Situation the Subsequent Step: Re-evaluate the remaining options, assuming the options from the earlier step are already identified. We’ll incorporate them within the subset of pre-selected options S within the shapley worth every time.
3. Repeat: Ask the mannequin: “Provided that the consumer already is aware of about Characteristic A, B, C, which remaining characteristic contributes probably the most data?”

By recalculating Shapley values (or marginal contributions) conditioned on the pre-selected options, we be sure that redundant options successfully drop to zero. If Characteristic A and Characteristic B are similar and Characteristic A is chosen first, Characteristic B not supplies new data. It’s robotically filtered out, leaving a clear, concise listing of distinct drivers.

Be aware: Yow will discover an implementation of this direct group and grasping iterative calculation in our Python package deal medpython.
Full disclosure: I’m a co-author of this open-source package deal.

Actual World Validation

Whereas this toy mannequin demonstrates mathematical flaws in shapley values technique, how does it work in real-life eventualities?

We utilized these strategies of Grouped Shapley with Winner takes all of it, moreover with extra strategies (which can be out of scope for this publish, possibly subsequent time), in advanced medical settings utilized in healthcare. Our fashions make the most of a whole lot of options with robust correlation that had been grouped into dozens of ideas.

This technique was validated throughout a number of fashions in a blinded setting when our clinicians weren’t conscious which technique they had been inspecting, and outperformed the vanilla Shapley values by their rankings. Every method contributed above the earlier experiment in a multi-step experiment. Moreover, our crew utilized these explainability enhancements as a part of our submission to the CMS Well being AI Problem, the place we had been chosen as award winners.

Conclusion

Shapley values are the gold normal for mannequin explainability, offering a mathematically rigorous solution to attribute credit score.
Nonetheless, as we now have seen, mathematical “correctness” doesn’t all the time translate into efficient explainability.

When options are extremely correlated, the sign is perhaps diluted, hiding the true drivers of your mannequin behind a wall of redundancy.

We explored two methods to repair this:

1. Grouping: Mixture options right into a single idea
2. Iterative Choice: conditioning on already offered ideas to squeeze out solely new data, successfully stripping away redundancy.

By acknowledging these limitations, we are able to guarantee our explanations are significant and useful.

If you happen to discovered this convenient, let’s join on LinkedIn