Beyond the Frame: Redundancy, Not Trajectory, Carries the Privacy Leak in Video Unlearning

The task

Make a video model forget something

A segmentation model labels every pixel of a video: person, car, road, sky. Class unlearning asks it to forget one label, say person, without retraining from scratch, so that the person can no longer be found in what the model outputs.

The catch

One frame looks fine, a few frames do not

Looking at one frame at a time, the person is gone. But if you average the model's outputs over about two seconds of video, a faint trace adds up and the person comes back. In our tests nearly every published method that keeps the model usable still leaks this way.

Our fix

Train against the averaging attacker

PACE trains the model against an attacker that does exactly this averaging, and adds a rule that stops the model from hiding the trace by freezing it in place. The person stays forgotten even when frames are pooled.

Why one frame is not enough

Each small frame below is mostly noise with a very faint blob in the middle, like a model whose output on any single frame looks clean. Slide to average more frames of the clip.

Single frames. No blob is visible in any of them.

Average, original order. The blob appears as more frames are added.

Average, shuffled order. Largest pixel difference from the left average: 0.0000

Frames averaged1

A toy picture with made-up numbers, not output from our models. It shows why averaging works on a faint but repeated trace, and why it does not care about frame order.

Watch it on a real clip

One validation clip from VSPW, 16 frames spanning about two seconds. Pick a model. The middle panel is what a frame-by-frame check sees. The right panel is what an attacker recovers by pooling the model's outputs over the clip.

The real frame. Red marks the person.

What the model predicts. Blue marks pixels it calls person.

What the attacker recovers. Brighter means more sure it is person.

Frame 1 of 16

The tiles are small because that is the resolution they were rendered at. This is one clip and one quick attacker run (250 training steps), so it illustrates the effect. The tables further down are the measured result.

All models side by side

What each model predicts, frame by frame

What the attacker recovers by pooling the outputs

The same clip as above, looping. Generated by scripts/make_qual_video.py.

Is it motion, or just repetition?

There are two things an attacker could use. The path the person takes through the video, or the same faint trace simply repeating in every frame. To tell them apart we scramble the frame order and run the attacker again. If the path mattered, scores would drop. They barely move, so repetition is what carries the leak.

frames in orderframes shuffled

Attacker score (average precision, out of 100) on VSPW person with a 3-frame clip. From the paper's table of adversary scores by model.

Left: averaging per-frame outputs recovers a silhouette. Right: time shuffling distinguishes trajectory from redundancy.

Three versions of the method

All three edit the same model and share the same training setup. They differ only in what the attacker is allowed to see and in the extra rule added to the penalty.

Video clipA few consecutive frames.

→

Model being editedStarts as the original model and is changed to forget the class.

→

Output mapsThe model's score for every class at every pixel, for every frame.

→

AttackerA small network that tries to find the forgotten class in those maps.

→

PenaltyThe model is penalized when the attacker succeeds.

Keep-it-working loss, in all threeThe model must still segment every other class correctly, and stay close to what the original model predicted for them. Pixels of the forgotten class are relabelled with their nearest neighbouring class.

PACE-F

attacker sees one frame

Attacker seesOne frame's output map at a time.

→

Per-frame attackerTries to find the class in that single frame.

→

PenaltyPenalized if the attacker finds the class in a frame.

In plain words: The model learns to make every single frame look clean.
Weak spot: A faint trace can remain and repeat in every frame. A one-frame attacker cannot see it, but averaging can.
In our tests: Still leaks on people: attacker score against a leak line of 35.66.

PACE-T

attacker watches the clip in order

Attacker seesThe whole clip, frames in order.

→

Clip attackerLooks across time for the class.

→

PenaltyAttacker success, plus a rule: the forgotten region must not move more than its surroundings.

In plain words: The model is trained against an attacker that uses several frames, and it is stopped from leaving a trace that jumps around.
Weak spot: The rule only punishes too much movement. A trace that is completely frozen costs nothing, and a frozen trace is exactly what averaging finds most easily.
In our tests: No leak found in all three settings: on people, on cars, on Virtual KITTI 2 cars.

PACE

adds an averaging attacker and a two-sided rule

Attacker seesThe clip in order, and also only the average of the clip.

→

Two attackersOne reads the ordered clip. One reads only the average, so frame order cannot matter to it.

→

PenaltySuccess of either attacker, plus a two-sided rule: the region must move about as much as its surroundings, no more and no less.

In plain words: The model is trained directly against the averaging attacker, and freezing the trace no longer escapes the penalty.
Cost: About 3.8% more training time than PACE-T (853.5 against 822.2 minutes on one GPU).
In our tests: No leak found in all three settings, with the lowest attacker score of the three versions: on people, on cars, on Virtual KITTI 2 cars. On Virtual KITTI 2 it keeps much less accuracy than PACE-T (7.6 against 21.1).

Results in plain words

Two questions for each method. Can an attacker still find the class? (lower is better, and below the black line means no leak found.) Does the model still do its job on everything else? (higher is better.)

Method	Attacker scoreaverage precision, lower is better	Model still worksmean IoU on the other classes, higher is better	In plain words

The black line is the leak line: the retrained model's score plus two of its trial standard deviations, the paper's rule for calling a model safe. The grey bar for the original model shows how much accuracy the model had before unlearning. Keeping under a tenth of the original accuracy is described as badly damaged, under a quarter as much lower, and under half as lower. That wording is this page's, not a rule from the paper. Numbers are copied from the paper's tables and each score averages two attacker trainings.

What to keep in mind

PACE costs accuracy on Virtual KITTI 2. There it keeps a mean IoU of 7.6 against 21.1 for PACE-T and 50.1 for the original, so PACE-T is the better trade-off on that dataset.

Two datasets and two classes. Real videos (VSPW, person and car) and synthetic driving videos (Virtual KITTI 2, car), all with one segmentation network, DeepLabV3 with a ResNet-50 backbone.

Short clips. The attacker sees about two seconds. We show that repetition, not motion, carries the leak at those lengths. We do not claim motion can never leak over longer stretches.

The video is an illustration. One clip and a quick attacker run. The measured results are in the tables.

One published method is an exception. On Virtual KITTI 2, FT scores below the leak line and keeps the model working, so it is not true that every usable method leaks.

Forgetting is not the same as safe. Several methods reach a low attacker score only by wrecking the model. A low score on its own is not a good result.