MODELING AND EXPLOITING VULNERABILITIES FOR DEEPFAKE DETECTION

The rapid progress of generative artificial intelligence has enabled the creation of highly realistic facial manipulations, commonly referred to as deepfakes. While such technologies offer significant benefits for media production and human-computer interaction, they also pose serious risks to privacy, security, and societal trust. Ensuring the authenticity of visual content has therefore become a critical challenge. Although recent deep learning-based detectors have demonstrated strong performance under controlled benchmarks, their reliability remains insufficient for real-world deployment, where manipulations are diverse, continuously evolving, and often imperceptible to human observers. Current deepfake detectors typically rely on deep neural networks that are trained in a supervised manner as binary classifiers that differentiate fake and real data.

Hence, they suffer from two main limitations. First, they are inevitably subject to overfitting issues and tend, therefore, to achieve poor generalization to unseen data variations. Specifically, their performance degrades significantly under unseen domains, high-quality forgeries, or real-world distortions. Second, deep architectures, including CNN and transformers, fail to capture by definition localized artifact-prone features, which typically characterize deepfakes, especially high-quality ones. Addressing these limitations requires moving beyond conventional binary classification toward models capable of modeling localized areas where generative processes are more likely to introduce subtle artifacts.

This thesis introduces a unified vulnerability-aware framework for deepfake detection across both images and videos. The central idea is to explicitly guide the learning process toward vulnerable regions that are defined as the areas that are more likely to incorporate blending artifacts. Since blending artifacts are common to most generation methods, this strategy enables detectors to focus on generic artifacts rather than patterns specific to a given dataset. At the frame level, a CNN-based framework, called LAA-Net, aims at modeling fine-grained vulnerabilities at the pixel level in deepfake images. This is achieved through multi-task learning strategies that jointly perform detection and artifact localization while integrating complementary multi-scale representations. This concept is further extended to transformer architectures through LAA-Former, which transfers vulnerability modeling from pixels to patches, unifying explicit local supervision and global relational reasoning within a single framework. As a result, it contributes to improving cross-domain generalization and robustness to noise while maintaining computational efficiency.

Extending this idea to video analysis, a vulnerability-aware spatio-temporal framework, namely FakeSTormer, is developed to capture intertwined spatial and temporal inconsistencies introduced during video synthesis. By disentangling the learning of temporal and spatial vulnerabilities, the proposed approach improves robustness against diverse high-quality and unseen deepfake videos.

Beyond model design, this thesis revisits the evaluation of deepfake detectors under realistic conditions. In fact, current evaluation protocols rely on AUC solely estimated on separate datasets. This cannot guarantee the maturity of deepfake detectors for real-world deployment, since strong AUC on each dataset in isolation can mask instability when data are mixed with strong distribution shifts. To address this issue, a new evaluation metric, termed Cross-AUC, is introduced to assess generalization stability across uncontrolled and mixed-domain scenarios.

Together, the proposed contributions advance the principled design, analysis, and evaluation of deepfake detection systems toward more generalizable, robust, and interpretable solutions, bringing automated deepfake detection closer to practical real-world applicability.