interpretability · spatial biology · 2026 · report

Do explanation tools find the right cells?

I built tissue where I know which cells matter, then asked the popular tools to point at them.

Code
github.com/tanmayhinge/tissue-graph-probing
Stack
Python, PyTorch, PyTorch Geometric

The problem

Show a model a slice of tumour tissue and it will predict something useful. The obvious next question is which cells made it say that. There are popular tools for answering this, and people use them to make biological claims.

The catch is that they get tested on real tissue, where nobody knows the true answer. So “the method highlighted the important cells” is a statement no one can check. It is taken on faith.

So I built tissue where I know the answer

Not real tissue. Cells with types and marker profiles, scattered under rules I control, with a label defined as a function of the cells themselves: a sample counts as positive when enough tumour cells have a T cell sitting close by. Because I write the rule, I can record every cell that contributed. Fig 1 shows six generated samples.

Six panels of synthetic tissue. The top row shows positive samples and the bottom row negative ones. Engaged tumour cells and the T cells enabling them are marked in red and dark blue, unengaged tumour cells in orange, decoy T cells in pale blue, with macrophages and stroma in grey.
Fig 1. Six generated samples, positives on top. The engaged region is the cluster where T cells sit on tumour; everything else is background or decoy.

Two extra things get recorded, and they turn out to matter more than the driver list. First, a stricter set: the smallest group of cells whose removal actually flips the label. Second, decoys, which are cells with identical marker profiles sitting in the wrong place. Anything worth calling an explanation should point at the first and ignore the second.

The test

Train a graph neural network on this tissue. It solves the task almost perfectly, around 0.999 AUC, which matters because you cannot ask what a broken model was thinking. Then hand each explanation tool a sample and ask which cells mattered, and score its answer against the recorded truth.

Most of them do not find the cells

Average precision at recovering the driver cells, by method Random guessing scores 0.058. GNNExplainer scores 0.044, below guessing. Attention rollout 0.090, saliency 0.095, integrated gradients 0.153. A marker heuristic that never consults the model scores 0.169, and CNN integrated gradients 0.189. CNN integrated gradients 0.189 marker heuristic (no model) 0.169 integrated gradients 0.153 saliency 0.095 attention rollout 0.090 random guess 0.058 GNNExplainer 0.044 average precision at finding the driver cells
Fig 2. Higher is better. The dashed line is random guessing.

GNNExplainer, probably the best known of these tools, lands at 0.044 against a random baseline of 0.058. It is worse than guessing. Attention rollout manages 0.090. The best graph method, integrated gradients, reaches 0.153.

The uncomfortable line is the highlighted one. That is a heuristic which asks “does this look like a tumour cell?” and stops there. It never touches the model. It scores 0.169, better than every graph explanation method tested.

The check that actually mattered

Here is a cheap test that is easy to skip. Scramble the network’s weights so it knows nothing, then run the explanation tools again. If a tool returns roughly the same answer on a randomised network as on a trained one, it was never reading the model in the first place.

GNNExplainer’s answers stayed 0.83 to 0.89 correlated on one architecture. Attention rollout reached 0.99. Both were, in effect, describing the shape of the graph rather than anything the model had learned. Only integrated gradients came through this cleanly.

One gotcha worth passing on

GNNExplainer randomisation correlation against optimisation budget Correlation with a randomised network falls from 0.852 at 30 epochs to 0.554 at 100, 0.366 at 200 and 0.283 at 400, then 0.340 at 800. The common default of 100 epochs sits partway down the slope. 0.2 0.4 0.6 0.8 30 100 200 400 800 common default optimisation epochs, log scale
Fig 3. Lower is better. The same test returns a different verdict depending on how long you run it.

That randomisation number is not a fixed property of the method. It depends on how long you let GNNExplainer optimise. At 30 epochs it reads 0.85, which looks damning. At 400 it settles near 0.28, which looks acceptable. The common default is 100, partway down the slope, where the number has not converged and means very little.

Anyone reporting this test should say what budget they ran it at. I have not seen that done.

Things I got wrong

My first generator was broken in a way that took a while to notice. A model could hit 93 % accuracy on it without looking at a single tumour cell, because the local density of T cells gave the answer away on its own. Every faithfulness number I would have computed on it was measuring a shortcut.

Then I got the GNNExplainer verdict wrong twice in opposite directions. I called it void, found the number was an artefact of too little optimisation, retracted that, and then over-corrected by generalising from the single most favourable setting. The final answer sits between the two. Both corrections are in the repository rather than tidied away, which I think is the honest way to leave it.

What this does and does not show

This is synthetic tissue and one family of labels. It does not prove these tools fail on real data, and a method that misses my driver cells might still point somewhere biologically useful. What it does show is that when the answer is checkable, most of these tools do not recover it, a heuristic with no model access competes with the best of them, and the control that catches this costs about ten minutes to run.

Full write-up, code and the gated test suite are in the repository.