Unsupervised Video Segmentation Algorithms Based On Flexibly Regularized Mixture Models
Claire Launay, Jonathan Vacher, Ruben Coen-CagliAbstract
We propose a family of probabilistic segmentation algorithms for videos that rely on a generative model capturing static and dynamic natural image statistics. Our framework adopts flexibly regularized mixture models (FlexMM) (Vacher 2022), an efficient method to combine mixture distributions across different data sources. FlexMMs of Student-t distributions successfully segment static natural images, through uncertainty-based information sharing between hidden layers of CNNs. We further extend this approach to videos and exploit FlexMM to propagate segment labels across space and time. We show that temporal propagation improves temporal consistency of segmentation, reproducing qualitatively a key aspect of human perceptual grouping. Besides, Student-t distributions can capture statistics of optical flows of natural movies, which represent apparent motion in the video. Integrating these motion cues in our temporal FlexMM further enhances the segmentation of each frame of natural movies. Our probabilistic dynamic segmentation algorithms thus provide a new framework to study uncertainty in human dynamic perceptual segmentation.
A communication paper for the 2022 ICIP conference can be found here.
Additional content
Here are four videos comparing the results of static and dynamic segmentation algorithms based on FlexMMs.
High temporal stability:
| Original image | Ground truth | Temp prop segmentation | Indep SIL1 segmentation |
 |
 |
 |
 |
 |
 |
 |
 |
Low temporal stability:
| Original image | Ground truth | Temp prop segmentation | Indep SIL1 segmentation |
 |
 |
 |
 |
 |
 |
 |
 |
If you have any question, feel free to send us an email, we'll be happy to answer them.