Semantic Attention Flow Fields for Monocular Dynamic Scene Decomposition

Abstract

We present SAFF: a dynamic neural volume reconstruction of a casual monocular video that consists of time-varying color, density, scene flow, semantics, and attention information. The semantics and attention let us identify salient foreground objects separately from the background in arbitrary spacetime views. We add two network heads to represent the semantic and attention information. For optimization, we design semantic attention pyramids from DINO-ViT outputs that trade detail with whole-image context. After optimization, we perform a saliency-aware clustering to decompose the scene. For evaluation on real-world dynamic scene decomposition across spacetime, we annotate object masks in the NVIDIA Dynamic Scene Dataset. We demonstrate that SAFF can decompose dynamic scenes without affecting RGB or depth reconstruction quality, that volume-integrated SAFF outperforms 2D baselines, and that SAFF improves foreground/background segmentation over recent static/dynamic split methods.

Method

Pyramid feature extraction

Semantic and saliency quality improves with our pyramid feature extraction approach.

The Balloon NBoard sequence below shows the low-resolution raw semantics and saliency from DINO-ViT alongside our higher-resolution versions. Note: features are projected via PCA, so some colour variation is expected—the colours are not directly comparable.

Feature volume integration

Semantic and saliency quality improves further through volume integration.

The Balloon NBoard sequence below shows the semantic and saliency input to the optimisation alongside the rendered output after optimisation.

Saliency-aware clustering

Saliency-aware clustering improves the decomposition.

The DynamicFace sequence below shows the clustering before saliency voting and before cluster merging.

Results

Supplemental decomposition results across six sequences. This page holds many comparison clips, so each dataset is collapsed by default and its videos load only when you open it. Every clip plays at half speed—the source renders run fast—with per-block play, reset, and speed controls.

Presentation

Download slides

Citation

@inproceedings{Liang2023SAFF,
  title     = {Semantic Attention Flow Fields for Monocular Dynamic Scene Decomposition},
  author    = {Liang, Yiqing and Laidlaw, Eliot and Meyerowitz, Alexander and Sridhar, Srinath and Tompkin, James},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2023}
}

Acknowledgements

We thank the computer vision community in New England for feedback, and acknowledge funding from NSF CNS-2038897 and an Amazon Research Award. Eliot Laidlaw was supported by a Randy F. Pausch '82 Computer Science Undergraduate Summer Research Award at Brown University.