MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F24%3A00379882" target="_blank" >RIV/68407700:21230/24:00379882 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1109/TIFS.2024.3409054" target="_blank" >https://doi.org/10.1109/TIFS.2024.3409054</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1109/TIFS.2024.3409054" target="_blank" >10.1109/TIFS.2024.3409054</a>
Alternative languages
Result language
angličtina
Original language name
MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection
Original language description
In this paper, we present MINTIME, a video deepfake detection method that effectively captures spatial and temporal inconsistencies in videos that depict multiple individuals and varying face sizes. Unlike previous approaches that either employ simplistic a-posteriori aggregation schemes, i.e., averaging or max operations, or only focus on the largest face in the video, our proposed method learns to accurately detect spatio-temporal inconsistencies across multiple identities in a video through a Spatio-Temporal Transformer combined with a Convolutional Neural Network backbone. This is achieved through an Identity-aware Attention mechanism that applies a masking operation on the face sequence to process each identity independently, which enables effective video-level aggregation. Furthermore, our system incorporates two novel embedding schemes: (i) the Temporal Coherent Positional Embedding, which encodes the temporal information of the face sequences of each identity, and (ii) the Size Embedding, which captures the relative sizes of the faces to the video frames. MINTIME achieves state-of-the-art performance on the ForgeryNet dataset, with a remarkable improvement of up to 14% AUC in videos containing multiple people. Moreover, it demonstrates very robust generalization capabilities in cross-forgery and cross-dataset settings. The code is publicly available at: https://github.com/davide-coccomini/MINTIME-Multi-Identity-size-iNvariant-TIMEsformer-for-Video-Deepfake-Detection.
Czech name
—
Czech description
—
Classification
Type
J<sub>imp</sub> - Article in a specialist periodical, which is included in the Web of Science database
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
<a href="/en/project/GM21-28830M" target="_blank" >GM21-28830M: Learning Universal Visual Representation with Limited Supervision</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Name of the periodical
IEEE Transactions on Information Forensics and Security
ISSN
1556-6013
e-ISSN
1556-6021
Volume of the periodical
19
Issue of the periodical within the volume
June
Country of publishing house
US - UNITED STATES
Number of pages
13
Pages from-to
6084-6096
UT code for WoS article
001252656400004
EID of the result in the Scopus database
2-s2.0-85195382276