Say an AI lays out a satellite image and rules that "this patch is burned ground." Usually there are two things we want to know. Where in the image did the model look, and which wavelengths of light did it use as its cue?
The trouble is that answering these two separately can produce explanations that don't line up. If you only look at which bands of light mattered on average across the whole image, you can end up citing light that was really only important for forest as the reason for a patch of burned ground. "Where" and "which light" are interlocked, and pulling them apart severs that link.
Satellite sensors capture light well beyond the red, green, and blue our eyes see, splitting it into several intervals. Each of these intervals is called a channel (also known as a wavelength band). And the channel that's decisive for forest is different from the one that's decisive for burned ground.
SSI (Spectral-Spatial Interpretation), from researchers at the University of Wisconsin–Madison, binds the two questions into one. It delivers, in a single table, "which channels contributed to the decision at each location." This table is called an attribution matrix. The rows are locations in the image, the columns are channels, and each cell is a score for "how much that light at that location contributed to the decision."
Here's roughly what it looks like (a conceptual example). Near-infrared and shortwave infrared are both invisible to the human eye — light with longer wavelengths than red.
| Location \ Light | Near-infrared | Shortwave infrared |
|---|---|---|
| Forest | High | Low |
| Burned ground | Low | High |
Even within the same image, forest locations respond to near-infrared while burned-ground locations respond to shortwave infrared. Capturing this difference in a single picture is SSI's core output.
Until now, explanation tools came in roughly two kinds. One is a map showing which locations matter. It shades the image by how strongly each region drove the decision (such a picture is called a heatmap). The other is a ranking of which light matters. It ranks which channels were important overall. (The specific technique names representing each family are collected in the appendix at the end.)
Both flatten one thing. The location map merges many channels into a single picture, and the light ranking averages each channel's effect across the whole image. Look at the table again. Forest locations responded to near-infrared, burned-ground locations to shortwave infrared. But if you average that table down its columns into a single row of "the light that matters overall in this image," the two lights blend into something muddled, and the very fact that forest and burned ground used different light gets erased. And that difference was exactly what we wanted to see.
There's a common shortcut here. It's a two-step scheme that multiplies the location map by a single global channel ranking — a so-called rank-1 structure. In effect, it applies the same channel-importance ratio to every location. But the same light can be decisive at one location and irrelevant at another. So the goal of attribution shouldn't be one global ranking, but a location-specific channel score that differs from place to place.
Say we want to explain one location in a burned region. SSI leaves the surrounding forest and city exactly as they were actually observed, and only touches the channels at that one location.
We need to be precise about "touches" here. It doesn't mean actually removing that light, but swapping it for a predetermined comparison value — a baseline. What you choose for this baseline changes the meaning of the question. Swap in the channel's mean value and you're asking, "What if the light at this location had been ordinary?"; swap in 0 and you're asking, "What if this light hadn't been there at all?" Those are different questions, so even for the same score you have to state which baseline was used. In this way you swap the channels at that location in various combinations and watch "how much the prediction changes with and without this channel."
The calculation that fairly divides up each channel's share is the Shapley value. Originally from game theory, it's a method for "dividing a jointly produced result among participants according to how much each contributed"; here, the channels are the participants. Because one channel's effect depends on which other channels are present alongside it, you don't look at a single channel in isolation — you average the change in prediction from adding that channel across many combinations and take that as its contribution.
The key device for cutting the scale down is here. Rather than solving for every (location, channel) across the whole image at once, when explaining one location you divide the contribution only among the channels at that location. If a location has C channels, you're solving many small C-player games, one per location, instead of one giant game. As the authors stress, they didn't invent a new division rule; the contribution is the problem setup itself — asking about a single location while holding its surroundings fixed at what was observed.
A useful property falls out of this design. If you sum all the channel scores at a location, it comes out exactly equal to how much the prediction changes when you swap that whole location to the baseline. In other words, it directly answers "how much does the result change if I erase this location wholesale?" But this sum only holds within a single location. It does not guarantee that you can add scores across locations and read the total as a contribution ratio over the whole image — a boundary we'll revisit in the "Limits" section below.
The closest point of comparison is the Owen value. Both use the same group structure of location-and-channel, but they diverge in how they handle outside context. The Owen value averages over the many ways the outside could have been and asks "on average, how much does this channel contribute," whereas SSI holds the outside fixed at this scene as observed and asks "in this image, how do the channels at this location contribute." It's not a matter of right versus wrong — the questions are simply different (§4.5 of the paper compares the two directly).
The evaluation was done on three datasets deliberately chosen to be different in character.
| Dataset | Input (channels) | Task |
|---|---|---|
| CUB-200-2011 | RGB, 3 channels, 200 bird species | Classification |
| EuroSAT | Sentinel-2, 13 bands | Land-cover classification, 10 classes |
| HLS Burn Scar | Harmonized Landsat–Sentinel-2, 6 bands | Segmentation |
(Model types and spatial-grid sizes are collected in the appendix table.)
The results read best grouped into two questions. (1) Does it find the information that drove the decision well, and (2) what does it cost to find that information?
(1) Does it pin down the evidence? The test works like this. The more the prediction shakes when you erase the locations and channels the explanation flagged as important, the more faithfully that explanation pinned down the model's actual basis (this family of metrics is called deletion/insertion — you erase or insert items starting from the ones flagged most important and watch how the prediction changes). SSI had the best such fidelity on all three datasets and under both kinds of baseline (mean and 0). For one example, under the per-channel mean baseline, CUB-200's insertion AUC was .7132, beating the space-only RISE's .6886, and on EuroSAT and HLS the gap widens.
Let's draw the interpretive line clearly. What this metric confirms is "how faithfully the explanation tracks the model's change in prediction," not "whether that signal is truly a cause that tells you about fire in the real world." What the model took as its cue and what nature actually tells you are not always the same. The figures for the remaining fidelity, spectral, and overlap metrics are collected in the table below.
| Evaluation | What it looked at | Result |
|---|---|---|
| ROAD (HLS) | Fidelity measured by erasing from most / least important | Best of 9 methods with LeRF AUC .657, lowest with MoRF AUC .1143 (both the good direction) |
| Spectral metrics (EuroSAT·HLS) | How well it pins down important channels | Highest Hit@K, lowest channel deletion AUC |
| Spatial overlap mIoU (HLS top 10%) | How much the channel-merged spatial score overlaps the ground-truth region | SSI .389 > DeepSHAP .277 > RISE .191; flat-Shapley KernelSHAP lags far behind even at the same granularity |
| Cost of the rank-1 assumption | Error of "same channel ratio at every location" | Smallest on CUB with only 3 channels, largest on EuroSAT with many channels |
(2) What does it cost? On HLS Burn Scar, SSI led the MC-Owen estimator across the attribution-quality metrics while using only about 6% of the evaluation budget. That's because it only samples the small C-player game at each location. That said, this 6% is a comparison against a single MC-Owen on one dataset, HLS, so it's hard to assert the same ratio would hold on data with far more channels.
Start with the trade-off the authors stated outright. SSI gives you location-conditional explanations and exact per-location aggregation, but in exchange it does not split the full prediction value across all locations with nothing left over. Because no rule can satisfy both at once (Proposition 1), they chose the former.
The boundary this choice creates in practice is sharp. You use it to look into what the basis for a decision was at a specific location, but you can't use it for a report that allocates a contribution ratio over the whole image — like "what percentage of this image did burned ground account for." If your use case needs "global accounting," where the sum of all location contributions matches the prediction value, SSI is not that tool.
The remaining constraints are tied directly to "under what conditions should the scores be read."
When the channel count C is small you can enumerate all subsets exactly (CUB-200 has C=3), but scalability when C grows large is beyond this article's scope (paper Appendix B.4).
ONTLE is a service that deploys AI project managers (PMs). One of the things that PM does is explain to the person in charge the basis for the automated decisions the system makes.
SSI itself is a technique for satellite imagery — not something you transplant into a business system code and all. But one question a PM should ask when reviewing an explanation comes straight from here. Say a risk-rating system classified some task as "high risk," and the person in charge asks the PM, "Why?" Here, the PM has just one thing to check. Is the explanation the person received a global average ranking ("delay signals are the most important overall"), or the basis for this particular decision ("for this task, the dependency status drove the rating")? A signal that is less important on average can be decisive for a specific case, so an explanation that stops at a single line of global ranking gets sent back with a demand for case-specific evidence. That's a practical principle the PM takes not from SSI's implementation, but from its problem awareness.
Technique names deferred from the main text - The map-showing-important-locations family (spatial attribution): Grad-CAM, RISE, and others. They show which regions drove the decision as a heatmap. - The ranking-important-light family (spectral attribution): attention weights, SHAP scores, and others. They rank which channel is important overall. - Differences from other families: flat SHAP–type methods treat every unit as a flat participant and can't capture the two-tier location-channel structure, while group Shapley sees a group only as a single lump and can't split within the group.
Theorems - Theorem 1 (local Shapley fairness): Within each location, SSI is the unique allocation satisfying local efficiency, within-location symmetry, within-location null player, and additivity. - Theorem 2 (contextual aggregation consistency): The sum of a location's channel attributions = the full-context contrast of swapping that whole location to baseline. It holds without assumptions of monotonicity, supermodularity, or a specific architecture. - Proposition 1: No rule can simultaneously guarantee "per-location context contrast = channel sum" and "global efficiency (fully distributing the total prediction value across locations)." SSI chose the former. - Proposition 3: Flat Shapley does not guarantee location-level aggregability. SSI's spatial sum is guaranteed by Theorem 2.
Metrics (all are tools that measure "how much the prediction changes when you erase or insert the information flagged as important") - joint deletion AUC: how fast the prediction drops when you erase in order of highest attribution. Lower is better (meaning you erased the important things first). - insertion AUC: when you insert in order of highest. Higher is better. - rank-1 deletion: a diagnostic for whether fidelity holds even when the matrix is approximated by its leading rank-1. - mIoU: how much the channel-merged spatial score overlaps the ground-truth annotation. Possible only on datasets with annotations. - Hit@K: the hit rate of matching the top channels against established spectral indices (NBR for burn scars, NDVI/NDRE for vegetation). Measured only on classes with reliable signals. - ROAD (LeRF/MoRF AUC): fidelity evaluation when erasing from most important / least important first.
Experimental setup, additional detail
| Dataset | Model | Spatial grid |
|---|---|---|
| CUB-200-2011 | ImageNet-pretrained ResNet-18, fine-tuned | Fine grid |
| EuroSAT | ResNet-18, all bands, trained from scratch | 8×8 |
| HLS Burn Scar | Lightweight UNet | 8×8 |
The comparison group is the dense/flat family (Integrated Gradients, DeepSHAP, KernelSHAP), a rank-1 perturbation control (RISE-based), and gradient methods on HLS (Expected Gradients, Guided IG, FullGrad). The RISE control is a decomposition map that multiplies a spatial score by a global channel score — a device that directly tests "is heatmap × global channel vector enough?" The baseline was the per-channel mean by default, and was rerun with 0 as well for a robustness check.