The question
Masked-sequence pretraining works, in the sense that it improves downstream performance across a large number of domains. What it learns is much less clear. A model that reconstructs masked regions of a spectrum could be recovering the physical process that produced the peaks, or it could be exploiting local smoothness and periodicity that happen to correlate with the physics.
Downstream accuracy cannot tell the two apart. They come apart under probing.
Setup
Synthetic data is a limitation for deployment and an advantage for this question, because it lets me ask what the encoder knows against a ground truth that is not itself estimated.
I built a frozen twelve-compound library with Lorentzian lineshapes and J-coupling multiplets
at Pascal-triangle intensity ratios. Mixtures of two to five components on a 2048-point ppm
grid, with additive noise, a rolling baseline, and peak jitter. Every mixture carries exact
ground truth for component identity, concentration, and the clean per-component signal. The
reconstruction identity mixture = sum(clean components) + baseline + noise is
verified to zero floating-point error on every sample.
Method
A 544k-parameter patch transformer: 64 patches, 4 layers, 4 heads. Pretrained with contiguous-span masking on 20,000 unlabelled spectra, disjoint from all labelled data. Held-out reconstruction error fell from 0.0089 to 0.0038. The model reconstructs denoised peak structure inside fully masked regions.
Result
The fine-tuning comparison came back close to null. Across label budgets, a from-scratch baseline catches up (Figure 4). Presence classification saturates.
Linear probes on the frozen encoders said the opposite. Concentration MAE was 0.150 for the pretrained encoder, 0.239 for a random encoder, and 0.138 for an encoder trained directly on labels. Roughly 0.14 macro-F1 gain on presence. The pretrained encoder never saw a label.
The two figures disagree, and that is the finding. The probe (left) says the pretrained encoder has organised real structure: concentration is nearly as decodable as from a label-trained encoder. The fine-tuning curve (right) says that structure buys nothing once the whole model is free to move on an easy task. Both are true.
What it means
The encoder had learned real structure. The null was therefore a statement about task difficulty and about the fine-tuning protocol, not about the representation. A task that a from-scratch model can solve within the available budget cannot distinguish a good initialisation from a bad one.
The prediction, and what happened
This page previously said the fine-tuning gap should reappear once the task was hard enough that a from-scratch model could not catch up in a fixed budget, and that if it did not, the probing result would need a different explanation. That sweep has now run: five difficulty levels interpolating signal-to-noise from 40 down to 6, with more overlapping components and more peak jitter, and a fresh encoder pretrained at each level so that a drop cannot be confused with a pretraining-and-test mismatch.
The prediction half held. At the hardest setting with 40 labels, pretraining is worth +0.082 macro-F1, every seed agrees on the sign, and the effect is about four times the seed-to-seed spread. It is the largest pretraining win anywhere in the project. But the tidy story does not survive contact with the paired per-seed numbers: the easy end also shows a consistent +0.040, so no gap when easy is simply false here, and the apparent gain at 160 labels is one seed carrying the average while another goes negative. I do not report that one as an effect.
The probes found something better than the prediction. As the task hardens, the pretrained encoder keeps beating a random one, and the margin widens. What breaks is the comparison to supervision. At the hardest setting a supervised encoder still reads presence at 0.921, and its concentration error barely moves across the whole sweep, while the pretrained encoder falls to 0.529.
The information survives the noise. The pretext task stops capturing it. The hard regime is not intrinsically unsolvable, because supervision demonstrably solves it on the same data. Whatever is failing is the objective, not the task.
The obvious explanation is wrong
The natural reading is that at a signal-to-noise ratio of 6 the model is spending itself predicting noise, since noise is exactly the part of the target that cannot be predicted. If that were true, the fix is to change what the model reconstructs.
So I changed only the target and held everything else fixed, reconstructing the noise-free component sum instead of the observed signal. That target is ground truth no real experiment would hand you, which is the point: it is an oracle, and it upper bounds what any better target could buy. It recovered 2% of the gap to supervision. At the easy end, where there is no noise to remove, the two are indistinguishable, exactly as they should be.
| difficulty | random | raw target | clean target | supervised | gap closed |
|---|---|---|---|---|---|
| easy | 0.875 | 0.933 | 0.936 | 1.000 | 4% |
| mid | 0.659 | 0.811 | 0.815 | 0.991 | 2% |
| harder | 0.532 | 0.704 | 0.720 | 0.984 | 6% |
| hardest | 0.374 | 0.529 | 0.539 | 0.921 | 2% |
The hypothesis I wrote down was wrong, and it is worth being specific about what that costs and what it buys. It rules out the follow-up I would otherwise have spent weeks on: a Noise2Noise-style objective is the genuinely self-supervised version of the same fix, and if the oracle buys nothing, no approximation to it will do better. What remains is a sharper question. My leading guess, and it is a guess, is that masked reconstruction is solvable locally: filling a hidden span only requires continuing the peak shapes suggested by the neighbouring context, never knowing which compound produced them. The objective would then never reward identity at all. Supervision holds up at 0.921 precisely because it optimises identity directly. Before believing any of that, there is a duller possibility to eliminate first, because it is a measurement artifact rather than a fact about learning: the probe reads mean-pooled patch tokens, so structure held nonlinearly, or concentrated in the CLS token, would be invisible to it.
Limitations
- The compound library is synthetic, frozen, and small: twelve compounds, distinct peaks, moderate noise, so the supervised task saturates quickly.
- One dimension. No real spectrometer data and no chemistry-grade realism.
- Three seeds per fine-tuning condition; the random-encoder baseline is a single draw, not an average over initialisations.
- In the sweep, one encoder per difficulty is shared across seeds, so the error bars cover fine-tuning and probe variance but not pretraining-run variance. A single unlucky pretraining run would be invisible to them, which matters given how much weight the +0.082 carries.
- The difficulty knob moves three things at once, so a change along it cannot be attributed to noise, overlap, or jitter individually.
- The reconstruction objective has a height bias that under-predicts tall peaks.
- Linear probes measure decodability, not whether a downstream model uses the structure.
Reproducibility
Config-driven YAML pipeline with typed dataclasses. Every entry point seeded. 46 tests, including one asserting that the difficulty knob actually changes the data rather than only the config. A logged decision record that keeps the wrong turns in. Every number on this page regenerates from one command.