Towards Untrusted Social Video Verification to Combat Deepfakes via Face Geometry Consistency
Workshop on Media Forensics @ CVPR 2020
Abstract
Deepfakes can spread misinformation, defamation, and propaganda by faking videos of public speakers. We assume that future deepfakes will be visually indistinguishable from real video, and will also fool current deepfake detection methods. As such, we posit a social verification system that instead validates the truth of an event via a set of videos. To confirm which, if any, videos are being faked at any point in time, we check for consistent facial geometry across videos. We demonstrate that by comparing mouth movement across views using a combination of PCA and hierarchical clustering, we can detect a deepfake with subtle mouth manipulations out of a set of six videos at high accuracy. Using our new multi-view dataset of 25 speakers, we show that our performance gracefully decays as we increase the number of identically faked videos from different input views.
Video
Slides
Dataset
About the data
The dataset is hosted by Brown University's library. We captured 24 participants speaking arbitrary sentences from six time-synchronised DSLR cameras. We then turned these input videos into a set of deepfakes by shuffling the audio around and synthesising new matching mouth motions via LipGAN. We also include facial landmarks processed from both the real and fake videos, which are the input to our video-matching code.
- Processed landmarks
- All data — the real and fake videos from 24 participants.
Please note: These data are for preliminary experiments only and we release them for scientific reproducibility only. While we have captured some diversity in human appearance across our 24 participants, this data is heavily biased. Nobody should expect experimental findings generated with these data to hold across wider populations, and nobody should train a machine learning model on these data and deploy it anywhere.
Code
We have two repositories:
- Python implementation of the core method. This was set up for a hackathon: the facial landmarks have been pre-extracted from the videos, as that step used an expensive preprocess.
- MATLAB implementation of the core method, plus all pre-processing code for the input videos and fake-video generation.
Citation
@inproceedings{Tursman_2020,
author = {Tursman, Eleanor and George, Marilyn and Kamara, Seny and Tompkin, James},
title = {Towards Untrusted Social Video Verification to Combat Deepfakes via Face Geometry Consistency},
booktitle = {2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
publisher = {IEEE},
year = {2020},
month = jun,
pages = {2784--2793},
doi = {10.1109/CVPRW50498.2020.00335}
}Eleanor also wrote an article on deepfakes and this approach in the ACM's student magazine—XRDS: Crossroads (DOI 10.1145/3416061).
Notes and Errata
- One participant did not wish for their data to be released. Results reported in the main paper are for 25 participants, not 24, and so reproductions of our results will vary slightly.
- Document issues: these have been fixed in our download PDF; the CVPRW proceedings PDF will still contain them.
- The Eq. 1 definition of the Mahalanobis distance was missing a transpose after the first parenthetical statement within the sum.
- In Eq. 2, the eigenvector matrix \(E\) is determined through PCA, which operates on the covariance matrix of the data matrix. So, for greater clarity, in the paragraph following Eq. 2, we have added the words 'of the covariance matrix' before 'of that same data matrix'.
- In Section 3.2, we have clarified that the clustering progresses via the minimum distance between leaves in any cluster, rather than any other linkage metric like an average over the cluster.