Probing a frozen rPPG encoder for identity under input degradation

Your face is blinking, very faintly

Every time your heart beats, it pushes a little more blood into the capillaries just under your skin. Blood absorbs green light more than the surrounding tissue does, so with each beat your face gets very slightly darker, then very slightly brighter again. Your eyes cannot see it. A camera can.

The technique is called remote photoplethysmography, usually shortened to rPPG. Take a video of a face, average the colour of the skin pixels in each frame, and you get a wobbly line that rises and falls at the rate of the heartbeat. That is genuinely all there is to the classical version of it. The wobble is tiny. In the video below, the average green value of the skin swings by about 1.0 of the 255 brightness levels a camera records, which is roughly a third of one percent. But it is periodic, and periodic signals are easy to pull out of noise.

Modern rPPG uses a neural network instead of an average, and it works better. You feed in a short clip of a cropped face and it outputs a pulse waveform. The selling point, when this gets built into a fitness app or a driver-monitoring system or a telehealth call, is that it is unobtrusive and privacy-preserving: no contact, no wearable, and the only thing that comes out is a heart rate.

That last claim is about the output. It is not a claim about the model's internal representation, the few hundred numbers the network computes on its way to producing the waveform. Those internal numbers, not the heart rate, are what gets cached, batched, sent to a server, or logged for debugging. Nobody in the applied literature reports what is in them. That is the gap this is about.

Three things are worth knowing before you invest in the rest of it. The heart-rate half did not work: the probe lands at 25.3 bpm against 28.8 for ignoring the video and guessing the dataset average, and its interval reaches that floor, so I cannot separate it from guessing. The identity half is measured across clips of one recording, so every number here is a ceiling on re-identification rather than an estimate of it, and none of them carries a confidence interval. And the encoder deserves less credit than the headline implies: raw pixels with no model behind them match it at every rung of the blurring. Each gets its own section below. What stays open is whether pulse and identity are genuinely coupled, or whether this encoder simply never transferred to synthetic faces.

Faces that belong to nobody

To ask the question you need face videos with known heart rates, which is normally an access-request-and-wait-three-weeks situation. I used SCAMPS instead: 2,800 videos of computer-generated avatars, rendered with a synthetic pulse driving the skin colour, so the true heart rate is known exactly rather than measured with a finger clip. Every face in this article is one of those avatars. None of them is a person. That is what makes it possible to publish the pictures at all, which is a slightly funny thing to have to say in an article about faces being identifiable.

Getting the data was the first surprise. SCAMPS ships a spreadsheet of per-file download links, and every one of them points at a storage account that no longer exists. The only working route is a single 593 GB compressed archive, and you almost certainly do not want all of it.

How you break a face on purpose

The experiment is a ladder. At the top, the model sees the full face crop, 72×72 pixels, with everything that is not skin blanked out. At each rung down, the skin region is averaged into progressively coarser blocks (32×32, then 16×16, 8×8, 4×4) until at the bottom there is a single average colour for the entire face, one number per colour channel per frame. Nothing about the pulse is deliberately removed on the way down. Averaging pixels together does not weaken a signal that every pixel shares; if anything it cleans it up.

Four more conditions sit off the ladder because they are not degrees of the same thing. Green only throws away the red and blue channels. Pixels shuffled scrambles the positions of the skin pixels, using the same shuffle for every frame in a clip, so each pixel keeps its own history intact while the face's layout is destroyed. Untouched keeps hair, clothing and background, as a reference point. And the last one I only added halfway through, for a reason that turns out to be the most interesting part of the result.

Those two are not arbitrary. Blurring and pixel shuffling are the ingredients of a published privacy method for exactly this setting: Gupta and Etemad's Privacy-Preserving Remote Heart Rate Estimation from Facial Videos (IEEE SMC, 2023) crops the face to cheek and forehead patches, shuffles the pixels under a per-video key held across frames as the shuffle here is, blurs with a 3×3 kernel, and reports facial recognition accuracy falling by over 60% while heart-rate error barely moves. The difference is which attacker gets measured. They embed the perturbed image with ArcFace and classify it, which is the right test for someone who intercepts the video, and no test at all for someone who gets the network's own embedding, the thing that actually gets cached and shipped off-device. A defence can be sound against the first attacker and untested against the second. Their own sweep also picks 3×3 because it is the optimal size for suppressing identification, meaning larger kernels start recovering identity again; on the ladder here blurring never suppresses it at any size. Neither result supports treating "blur it more" as a privacy control.

Two questions, one set of numbers

Here is the whole method. Take a released heart-rate model. Freeze it: no training, no fine-tuning, the weights never move. Push a clip through it and grab the activations from three layers inside, which gives 1,536 numbers per clip. Then fit the simplest possible thing on top of those numbers, twice: once to predict the heart rate, and once to predict which person this is, from a fixed set of 120. If a plain linear classifier can read something off the embedding, the embedding contains it. This is called probing, and its whole appeal is that a linear model is too weak to invent structure that is not there.

The catch with probing is that a number on its own is meaningless. Ninety percent accuracy sounds excellent until you learn that ninety percent of the examples were the same class. So every result here is reported against four floors, and none of them is optional:

An untrained model (the identical architecture with random weights, never trained on anything) tells you what you would get from any arbitrary projection of the input. Raw pixels (the same probe fitted straight onto the degraded image, with no model at all) tells you whether the model was needed. Shuffled labels tell you what the probe scores when there is definitionally nothing to learn. And POS and CHROM, two hand-derived formulas from 2013 and 2016 that involve no learning whatsoever, tell you whether a decade of deep learning bought anything.

The face survives being destroyed

120 people, 480 clips. Guessing at random gets you 0.83%.

Identity is 100% at the full crop, which is not surprising. It is still 100% at a 4×4 block average. Sixteen coloured squares per frame, one of 120 possible people, and the probe does not make a single mistake. The entire ordered ladder, every rung from a full-resolution face down to sixteen blocks, costs essentially nothing in re-identifiability. These are point estimates over 120 videos with no confidence intervals attached, which is a real gap and one I come back to at the end.

Shuffling the pixels does not help either: 99.2%. Scrambling where everything sits on the face leaves the person just as identifiable, which already hints that the model is not relying on facial features in the way you might picture.

Before going further, the shuffled-label control: run the identical probe with the identities randomly reassigned, and it scores 0.8%, which is chance to three decimals. So the 100% is not a probe with enough capacity to memorise its way to any answer you like. Give it nothing, and it finds nothing.

The outline was giving it away

Look again at the bottom-left panel of Fig 2: the 1×1 condition, one flat colour for the whole face. Identity there drops to 75.8%, which is a real drop, but it is still a hundred times chance. From one colour? That made no sense until I looked at the picture properly.

It is not one colour. It is one colour in the shape of that person's face, with holes where their eyes and eyebrows are, because everything outside the skin region gets blanked to black and the skin region is a silhouette of them. I had built a condition that removed all the texture and left a perfect stencil of the subject. The mask was leaking the identity the pixels no longer could.

So I added one more condition: take that single average colour and paint it across the entire frame, no outline at all. Three numbers per frame, and nothing else. Identity falls to 12.5%.

That is the actual finding, and it is a more useful one than the tidy curve I set out to draw. Blurring does not anonymise a face. Blurring plus deleting the shape of the face gets you most of the way there, and even then, 12.5% out of 120 people is about 15× what you would get by guessing.

There is a wrinkle here worth sitting with. It shows up on the verification metric, which asks a different question from the one above: shown one matching and one mismatched pair of clips, how often does the embedding rank the true match higher? That is an AUC, so 0.5 is a coin flip and 1.0 is never wrong. On it, the untrained network, the one with random weights that was never trained on anything at all, scores 0.942 on this condition, against the trained model's 0.726. A randomly initialised encoder is better at telling these people apart than the one that spent its training learning to read pulses. Whatever survives here is a property of the input, not something the heart-rate objective went looking for. You do not escape it by choosing a different model.

The encoder throws away what its input keeps

The raw-pixel control has matched the embedding at every rung so far, both of them flat at 100% the whole way down. At the bottom it stops matching, and it stops in the direction nobody expects. With the outline deleted, the embedding falls to 12.5% while the same probe on the identical input, with no network in the loop at all, holds 98.3%. On the verification metric that is 0.726 against 1.000. Passing the image through the encoder destroyed identity that survived the degradation itself.

The likely reason is mundane, and it is a guess rather than a result because nothing here isolates it: both checkpoints are the DiffNormalized variants, so the network is fed frame differences, and a flat coloured frame has almost none. The constant skin tone is normalised away before the first convolution, while raw pixels keep the absolute colour that on 120 rendered avatars is very nearly a fingerprint. Either way it cuts against the tidy version of the privacy story. Encoders are not uniformly worse than the pixels they consume, and a defence that reports what an image-space attacker can recover is not measuring the same thing as one that reports what survives in the representation.

It is not one checkpoint being strange

A result this lopsided invites the obvious objection: maybe this particular set of weights is peculiar. So the whole sweep also ran on a second released checkpoint: the same architecture trained by the same people on UBFC-rPPG, a different dataset of real subjects, with no shared training data and no shared fitting. If the collapse at the last rung were an artefact of one model, the two would not agree about where it happens.

They agree. Both hold at a perfect score through the coarse block averages, both slip at the single trace, and both fall off a cliff at exactly the same place, the moment the silhouette goes. The two disagree on the depth of the fall (12.5% against 24.2%) but not on where it happens or what causes it. Two models, trained on different people, degrade the same way, because what is being destroyed is in the input rather than in either of them.

The heart rate, meanwhile, was never really there

Now the part that does not flatter the experiment. The bottom panel of Fig 4 is nearly flat, which I could present as "heart rate survives the degradation." It would be a fair description of the shape and a misleading description of the result, because the line is flat at a level that is not much of an achievement.

If you ignore the video entirely and always answer with the dataset's average heart rate, you are wrong by 28.8 beats per minute. The probe on the frozen embedding manages 25.3. POS, the 2013 formula, gets 15.8 on the same clips, and CHROM, its 2016 successor, gets 21.5. So on this data the embedding carries the person's identity completely and their pulse only faintly.

And the 25.3 should not be quoted without its interval. Bootstrapped over videos, it runs from 22.1 to 28.7 beats per minute. The top of that range is 28.8, the number you get for free by ignoring the video. The model's advantage over guessing is not something this study can distinguish from zero. That sentence is less satisfying than a curve, and it is what the data supports.

The likely reason is that this particular model was trained on videos of real people and I ran it on synthetic renders. Faces built by a graphics engine are not the same distribution as faces lit by a real room, and the pulse is the part that would suffer first: it is a sub-one-percent colour fluctuation, and a renderer approximates it. Identity, which lives in gross appearance, transfers fine. The honest summary is that this is a strong result about re-identification and a weak one about heart rate.

The setting that does not exist

All of that was in service of one question, and it is worth stating in the form it was asked, before any of the numbers existed. If you are shipping a camera that reads pulses and you want it to be honest about privacy, you need a rung of this ladder you can actually run at: one that keeps the heart rate roughly as good as the full-resolution crop, and pushes re-identification down to something close to a coin flip. Written down in advance, that was within 2 beats per minute of the full crop, and verification AUC at or below 0.55, which on the scale above is a hair better than a coin flip.

Here is the part that decides it. Every condition in the study, ladder and off-ladder alike, comes in under the heart-rate tolerance: the pulse never got appreciably worse, because there was not much of it there to lose. So the whole question reduces to the second criterion, and that one is not close.

Nothing comes near it. The best any rung of the ordered ladder manages is 0.898, and even going off the ladder entirely, deleting the outline down to three numbers per frame, only reaches 0.726. The target was 0.55. reports/operating_point.json in the repository records the answer as null, which is the most useful thing this study produced: on this dataset, with this encoder, there is no amount of blurring that hides the person while keeping the pulse. You are not trading one for the other. You are losing the pulse first and keeping the person all the way down.

Before that gets quoted as a result, here is the reading that would undo it. An empty target region is consistent with two very different worlds. In the first, pulse and identity really are entangled in this representation, and no degradation separates them. In the second, this encoder never worked on these faces at all, the heart-rate axis was dead on arrival, and "the pulse does not survive degradation" is just a restatement of the fact that there was no pulse in the embedding to begin with. The heart-rate numbers are equally consistent with both, which is exactly what makes it worth saying out loud.

One piece of evidence separates them partway. POS, a formula from 2013 with no parameters and no training at all, recovers heart rate from these same videos at 15.8 bpm, so the pulse is genuinely present in the pixels and nothing about SCAMPS makes it unrecoverable. What failed is the frozen encoders' transfer onto rendered faces, not the data. That narrows the second world without closing it, because an encoder that cannot read the pulse in the first place cannot be asked to trade it away. Settling it needs a real-data arm where these checkpoints are in-domain: if the target region is still empty once the heart-rate axis is alive, the coupling claim stands, and if a rung opens up then this study measured a broken encoder and said something about the ladder instead. That run has not happened yet. Until it does, the identity result stands on its own and the tradeoff claim is provisional.

What I would not claim from this

The identity task here is easier than real re-identification. SCAMPS gives no indication of which avatar is which across videos, so "the same person" means "a different four-second clip of the same twenty-second recording." Same avatar, same lighting, same camera. A probe that tells those apart has done something easier than recognising someone who walked back in on a different day. The number is an upper bound, and the datasets that would make it a fair estimate (PURE, with ten subjects recorded in six sessions each, and MMPD, with thirty-three subjects across four lighting conditions) are the obvious next step. The code is already written against an interface with those two in mind.

The headline identity numbers are also bare point estimates. The heart-rate results carry bootstrap intervals over videos (that is where the 22.1 to 28.7 above comes from), and the identity results do not, which is the wrong way round: the claim I am leaning on is the one without an interval, and the weaker claim is the one that has it. A 100% out of 120 and a 12.5% are far enough from their floors that an interval would not change the reading, but "far enough that it would not matter" is a judgement, not a measurement, and it should be a measurement.

And SCAMPS is synthetic. Skin tones in it are whatever the renderer produces, not whatever the world contains, so nothing here says anything about how this behaves across real skin tones. That question needs MMPD, which labels Fitzpatrick group, and the analysis code refuses to pool heart-rate error across those groups when they are present. Reporting one average across skin tones is exactly how you hide that a method works for some people and not others.

Three things that were wrong first

Every heart rate was a multiple of 14.06. The toolbox's own frequency estimator sizes its transform to the length of the clip, which for a four-second clip at 30 frames per second gives bins 14.06 beats per minute apart. Every value it returned was 126.56, or 98.44, or 70.31, which are 9, 7 and 5 times 14.0625. A study about a few beats per minute cannot be measured on a grid that coarse, and it took plotting the numbers and noticing they were suspiciously round to catch it. Padding the transform fixes it; the test suite now asserts that the padded version still reproduces the original exactly when the padding is switched off.

One avatar was wearing a VR headset. The face detector, a twenty-year-old cascade classifier, reasonably found no face, and then instead of failing locked onto a 60×60 patch of ornamental wallpaper on the back wall and reported success. The pipeline dutifully cropped the wallpaper, resized it to 72×72 and fed it in. Its skin mask was empty, so nothing downstream would have crashed: the averages would have been zeros and the probe would have quietly fitted on a row of blanks. It is now caught by a check that the crop actually contains skin, and that video is excluded with its reason recorded.

The machine spent an hour swapping. The sweep held every clip's frames in memory at once, about a gigabyte on a laptop with eight, and drove itself four gigabytes into swap. The fix was one line: keep the frames as views into the memory-mapped files on disk instead of copying them out. The stage that had been running at 1.4 seconds per clip dropped to 45 milliseconds. It is a reminder that "why is this slow" is worth thirty seconds of measurement before it is worth any redesign.

Reproducing it

The code is at tanmayhinge/rppg-identity-tradeoff. It is config-driven and seeded, it runs on an M2 laptop in about two hours once the data is down, and the test suite is 107 tests with the degradation ladder and the probing logic as the two acceptance gates. The preprocessing is checked to be bit-identical to the reference toolbox's, so "we reimplemented it" never quietly became "we changed it".

My favourite test is the cheapest one. POS and CHROM only ever see the average colour of the skin, and every rung of the ladder is built to preserve that average exactly. So both must return identical heart rates at every rung. They do, to the digit, on real data. If the masking, the block partition, the alignment between mask and frame, or the clip assembly is wrong anywhere, that equality breaks. One assertion covers most of the pipeline.

Credit

The avatar images are from SCAMPS: D. McDuff, M. Wander, X. Liu, B. L. Hill, J. Hernandez, J. Lester and T. Baltrusaitis, SCAMPS: Synthetics for Camera Measurement of Physiological Signals, NeurIPS 2022, used under the Research Use of Data Agreement v1.0, which permits the small excerpts of the data needed to explain work done with it. The frozen models, the preprocessing and the POS and CHROM implementations come from rPPG-Toolbox: X. Liu et al., rPPG-Toolbox: Deep Remote PPG Toolbox, NeurIPS 2023. This is non-commercial research use.