Xiaomi launches new metrics for video object removal without reference

The MiLM Plus team at Xiaomi released PROVE, an evaluation suite that seeks to bridge the gap between the true visual quality of object-removal models and what existing metrics capture. Diffusion models can convincingly reconstruct shadows, reflections and hidden structures, yet PSNR, SSIM, LPIPS, ReMOVE and CFD rank their outputs inversely. The discrepancy is structural: removal is an ill-posed task with many valid solutions, so there is no single ground-truth for comparison. PROVE, accepted to ACM MM 2026, introduces two perceptual metrics—RC-S for spatial coherence and RC-T for temporal consistency—along with PROVE-Bench, a two-layer real-world video benchmark. Both metrics operate locally on the edited region and do not require a reference video.

Why the legacy metrics fail

The authors identify three systematic failure modes. Full-reference bias: PSNR, SSIM and LPIPS assume point-to-point correspondence, thus rewarding copy-paste rather than genuine removal; residual shadows occupy only a few pixels and receive minimal penalty. Moreover, truncating diffusion inference steps can improve PSNR and SSIM while visual quality collapses, effectively regressing to the mean. No-reference blindness: on ROSE-Bench, progressively blurring the marked region leaves both ReMOVE and CFD scores unchanged, eventually scoring higher than the unblurred baseline. CFD’s SAM-based component also raises false alarms: a correctly reconstructed bicycle seat is flagged as hallucination and receives a lower score than the original input. Temporal insensitivity: Temporal Consistency and Temporal Flickering metrics operate on full-frame features; because the edited region is a small fragment of the frame, injecting Random Drop or Random Replace corruptions on DAVIS barely moves the scores, sometimes in the opposite direction.

How RC-S and RC-T work

Both metrics share a core idea: align feature-space distributions locally rather than aggregating globally. RC-S (spatial) runs connected-component analysis to split the mask into independent objects. Each bounding box is expanded by one-third of its side length, the crop passes through DINOv2, and the mask is resampled to the feature resolution. A w × w sliding window scans the feature map and computes the squared Maximum Mean Discrepancy with a Gaussian RBF kernel between masked features and locally sampled background features. Scores are averaged per object and then across all objects. RC-T (temporal) merges masks of adjacent frames under a union operation to avoid misalignment, then applies the same MMD computation along the temporal axis.

Deployment: a CI tool, not a product feature

PROVE is released as an Apache-2.0 Python repository with a single CLI entry point (run_prove_metrics.py). Requirements are Python 3.10+, PyTorch 2.6+, Transformers 4.51+ and DINOv2-giant weights. Masks are mandatory; white pixels indicate the deleted object. RC-S runs in 134.6 ms per frame on a single RTX 4090, making nightly CI gates practical even for startups; organizations with large editing catalogs benefit most because no paired ground truth is needed. Intended uses include model bake-offs, CI gating with RC-S/RC-T, tuning inference steps or quantization without reference, filtering training data, and shaping reward models. The metrics are not suited for real-time on-device scoring or for cases where large shadows or reflections extend beyond the cropped evaluation region.

Target audience and scope

The primary audience comprises computer-vision researchers and applied AI/ML engineers working on editing pipelines; secondary audiences are MLOps engineers integrating the scores into CI and product managers conducting vendor comparisons. Relevant industries include smartphone gallery and camera apps, short-form video editing, e-commerce catalog cleaning, advertising and media archives, post-production cinema/VFX, real-estate visualization, and privacy-preserving mapping archives. The three documented failure modes remain pertinent: the task stays ill-posed and one-to-many, so there is no single correct solution; PROVE merely provides a scale that speaks the language of human perception rather than raw pixels.