SHAP & LIME

A gradient-boosted model declines a loan application and the applicant asks why. "Feature importance" cannot answer that question — it describes the model's habits across everyone, and this is one person. Both methods here answer the harder question: for this row, how much did each feature move the prediction? One computes it exactly, the other guesses it from a local fit, and they do not always agree.

SHAPShapley valuesLIME local vs globalmodel auditing

The model, and why its answer is not a sentence

The model below scores default risk from four features: credit utilization, late payments in the last two years, income, and years of credit history. It is deliberately non-additive — utilization and late payments compound each other, and income only protects you when your utilization is low. That is not a contrivance; it is what tree ensembles learn on real credit data, and it is exactly why you cannot read a coefficient off the model and call it an explanation.

The scoring function here is written down rather than fitted, so every value on this page is exact and reproducible rather than an artefact of one training run. Nothing else changes: SHAP and LIME both treat the model as a black box that maps four numbers to a score, which is the only thing they ever assume. On a real GradientBoostingClassifier you would reach for TreeExplainer for speed — the question it answers is the one being answered below.

Across the whole book the risk score averages 30.12%. That number is the anchor everything below is measured against: an explanation is the account of how this applicant got from the average to their own score. The panel opens on an applicant who scores 30.13% — a hair above the average, which looks like the model having no opinion at all. Look at the bars before you believe that.

SHAP: split the gap between the average applicant and this one

applicant thin file

attribute using all 24 orderings (Shapley)

What "the feature wasn't there" has to mean. To ask how much utilization contributed, you need the model's output without it — but the model has no such mode; it demands four numbers. SHAP substitutes values drawn from a background population and averages the result: "what would this model have said if I knew everything about this applicant except their utilization?" So the background dataset is not a technicality. Change who you compare against and every number on this page changes. Explain a subprime applicant against a prime background and the model will look like it is punishing them for things that are ordinary in their own segment.

Why average over orderings at all?

Suppose you reveal the applicant's features to the model one at a time and record how much the prediction jumps at each step. That gives you an attribution — but a different one for each order you pick, because interacting features steal each other's credit. Reveal utilization first and it takes the payment record's share; reveal the payment record first and the reverse happens. Scrub the second slider above and watch it happen.

Shapley's answer is to refuse to choose: average the marginal contribution over all n! orderings. With four features that is 24 orderings and this page computes every one of them exactly. With forty features it is 8×10⁴⁷, which is why the real shap library samples orderings (KernelSHAP) or exploits tree structure to get the same answer in polynomial time (TreeSHAP) — and why TreeSHAP being both exact and fast is most of the reason SHAP won on tabular data.

The property that makes it auditable. Shapley values are the only attribution satisfying all four of: efficiency (they sum exactly to prediction − base), symmetry (features that always contribute identically get equal credit), dummy (a feature the model ignores gets exactly zero) and additivity (explain an ensemble by summing its trees' explanations). Efficiency is the one a regulator cares about: the numbers add up, so there is no unattributed residue you could hide a proxy for a protected class inside. The panel checks it live — the "sum" readout is Σφ + base against the model's actual output.

LIME: fit a simple model near this point and read its coefficients

LIME takes the opposite route. Rather than reason about the model, it perturbs the applicant a few hundred times, asks the model what it thinks of each variant, weights those variants by how close they stayed to the original, and fits a weighted linear regression to that cloud. The linear model's coefficients are the explanation. It is model-agnostic and fast, and its central knob — how wide "close" means — is the thing nobody tunes.

LIME against SHAP on the same applicant

applicant safe

kernel width σ — how big is "nearby" 1.50

The fidelity score does not save you. The obvious defence is "check that the local surrogate fits well" — LIME reports a weighted R². Watch it while you drag σ: for the safe applicant, utilization's attribution runs from −4.4 to −15.7 across the range, a factor of 3.57, while R² only ever moves between 0.732 and 0.832 — and not even in the same direction, since its worst value is at σ = 0.40, in the middle. Every one of those explanations fits its perturbed cloud about equally well. A high R² tells you the surrogate described the neighbourhood you chose; it says nothing about whether that was the right neighbourhood.

Nor does agreement on the headline feature vouch for the rest. Slide the applicant to maxed out at the default σ = 1.50: on utilization the two methods are effectively identical (+50.8 against +50.1), and on late payments LIME books +9.7 where SHAP says +13.0 — three quarters of the value. A linear surrogate has no way to represent the util×late interaction, so it hands the shared credit to whichever feature is easier to fit and shortchanges the other. SHAP splits that interaction between them by construction. If the second reason on an adverse-action notice is the one being under-reported, "the top feature matched" is not the reassurance it sounds like.

Which do you reach for?

Use SHAP when the explanation is going to be defended — credit, insurance, hiring, anything with an appeals process. It has guarantees, it is deterministic, it sums, and for trees it is exact and fast. On a boosted model, TreeExplainer is close to free.
Use LIME when the model is a black box you cannot instrument, especially for text and images, where the perturbation ("delete this word", "grey out this superpixel") is meaningful in a way that substituting a background value is not.
Neither is causal. Both explain the model, never the world. "Utilization contributed +46 points" means the model leans hard on utilization, not that paying the card down would lower this applicant's real risk by 46 points. Every attribution here is a statement about a fitted function, and treating it as an intervention is how explainability projects end up recommending things that do nothing.
Correlated features degrade both. SHAP's background substitution builds applicants that could not exist (income 25k, home value 900k) and asks the model about them; LIME's perturbations do the same. Attribution then gets shared arbitrarily among correlated columns — which is why "drop the low-SHAP features" is a bad feature-selection rule.

Take away: global importance ranks the model's habits; SHAP and LIME account for one decision. They can disagree with each other, they both depend on a choice you probably left at its default — the background set, the kernel width — and neither one tells you what would happen if the applicant changed something. Report them with the choice that produced them attached.
Next: the Phase 3 boss challenge — build, evaluate and defend a model end to end.