Disentangling mixed spectra

Does masked-sequence pretraining learn physical structure, or does it learn the noise?

The question

Masked-sequence pretraining works, in the sense that it improves downstream performance across a large number of domains. What it learns is much less clear. A model that reconstructs masked regions of a spectrum could be recovering the physical process that produced the peaks, or it could be exploiting local smoothness and periodicity that happen to correlate with the physics.

Downstream accuracy cannot tell the two apart. They come apart under probing.

Setup

Synthetic data is a limitation for deployment and an advantage for this question, because it lets me ask what the encoder knows against a ground truth that is not itself estimated.

I built a frozen twelve-compound library with Lorentzian lineshapes and J-coupling multiplets at Pascal-triangle intensity ratios. Mixtures of two to five components on a 2048-point ppm grid, with additive noise, a rolling baseline, and peak jitter. Every mixture carries exact ground truth for component identity, concentration, and the clean per-component signal. The reconstruction identity mixture = sum(clean components) + baseline + noise is verified to zero floating-point error on every sample.

Figure 1One mixture drawn over its clean component signals. To be exported from scripts/visualize_mixtures.py.

Method

A 544k-parameter patch transformer: 64 patches, 4 layers, 4 heads. Pretrained with contiguous-span masking on 20,000 unlabelled spectra, disjoint from all labelled data. Held-out reconstruction error fell from 0.0089 to 0.0038. The model reconstructs denoised peak structure inside fully masked regions.

Figure 2A masked span with the reconstruction overlaid on ground truth. To be exported from the pretraining run.

Result

The fine-tuning comparison came back close to null. Across label budgets, a from-scratch baseline catches up (Figure 4). Presence classification saturates.

Linear probes on the frozen encoders said the opposite. Concentration MAE was 0.150 for the pretrained encoder, 0.239 for a random encoder, and 0.138 for an encoder trained directly on labels. Roughly 0.14 macro-F1 gain on presence. The pretrained encoder never saw a label.

Fig 3. Probe reads structure out of the frozen encoder.

Fig 4. Fine-tuning erases the difference.

The two figures disagree, and that is the finding. The probe (left) says the pretrained encoder has organised real structure: concentration is nearly as decodable as from a label-trained encoder. The fine-tuning curve (right) says that structure buys nothing once the whole model is free to move on an easy task. Both are true.

What it means

The encoder had learned real structure. The null was therefore a statement about task difficulty and about the fine-tuning protocol, not about the representation. A task that a from-scratch model can solve within the available budget cannot distinguish a good initialisation from a bad one.

What would falsify it

The fine-tuning gap should reappear once the task is hard enough that a from-scratch model cannot catch up within a fixed budget. A robustness sweep over signal-to-noise ratio and peak overlap is running now. If the gap does not reappear, the probing result needs a different explanation, and I will write that here.

Limitations

The compound library is synthetic, frozen, and small: twelve compounds, distinct peaks, moderate noise, so the supervised task saturates quickly.
One dimension. No real spectrometer data and no chemistry-grade realism.
Three seeds per fine-tuning condition; the random-encoder baseline is a single draw, not an average over initialisations.
The reconstruction objective has a height bias that under-predicts tall peaks.
Linear probes measure decodability, not whether a downstream model uses the structure.

Reproducibility

Config-driven YAML pipeline with typed dataclasses. Every entry point seeded. 31 tests. A logged decision record. Every number on this page regenerates from one command.