Predicting Ideology from Facial Photos Using Deep Learning: Expressions, Attractiveness, and Extra-Facial Information

Key Chapter Title List

1. Faces and Ideology
2. This Study
3. Data and Code Availability Statement
4. Sample
5. Image Preparation
6. Measurement Metrics
7. Analytical Methods
8. Heatmaps and Extra-Facial Information
9. Predicting Ideology Using Neural Networks
10. Identifying Salient Features
11. Discussion
12. References

Document Introduction

In the digital age, deep learning technologies can predict sensitive personal information from public data such as facial photographs. However, the core source of information enabling these successful predictions remains unclear. This ambiguity not only limits the public's ability to guard against privacy leakage risks but also affects the scientific application value of deep learning results. This study focuses on the sensitive topic of political ideology. The central question is: what information in facial photographs supports algorithmic predictions of ideology, and are these predictions associated with known facial morphology or expression features?

The research uses a sample of 3,323 Danish politicians (including local election candidates and members of parliament). It integrates multiple techniques such as Convolutional Neural Networks (CNN), heatmap visualization, facial expression coding, and classification of identifiable features like masculinity and attractiveness. This approach moves beyond the limitations of traditional studies that focus solely on prediction accuracy, delving deeper into the mechanisms behind the predictions. Sample selection followed strict screening criteria, excluding confounding factors such as non-European ethnicity and low-resolution images. Model reliability was enhanced through data augmentation and optimization of pre-trained networks.

The study employs a multi-stage analytical approach: First, an ideology prediction model is constructed using the VGG16 pre-trained network, with separate training, validation, and testing for male and female samples. Second, heatmap technology is used to locate key areas in the images crucial for prediction, distinguishing the influence of facial information from extra-facial information. Finally, through methods like facial synthesis and correlation analysis, the study examines the association between expressions (e.g., happy, neutral), morphological features (e.g., facial width-to-height ratio, attractiveness), and the model-predicted ideology. Data processing strictly adheres to Danish and EU GDPR regulations. All sample images are sourced from publicly available resources of the Danish Broadcasting Corporation, ensuring the study's compliance and reproducibility.

Key findings show that the neural network achieved a prediction accuracy of 61% for both male and female samples, significantly higher than random guessing. Heatmap analysis revealed that predictions for female samples primarily relied on facial features, whereas predictions for male samples were initially confounded by extra-facial information (e.g., collar features). After removing this information, the accuracy for males dropped to 61% (from an original 65%). Regarding facial expressions and morphological features, a happy expression showed a positive correlation with conservative ideology, while a neutral expression showed a negative correlation. Among female samples, attractiveness scores showed a significant positive correlation with the model-predicted conservative tendency. For male samples, masculine features (facial width-to-height ratio) showed no clear association with ideology.

As an application example of "Explainable Artificial Intelligence" in the social sciences, this study is the first to confirm an association between model-predicted ideology and independently classifiable facial features. It provides an empirical basis for understanding the potential link between facial information and ideology and offers important insights for privacy protection policy formulation—non-explicit features in facial photographs may constitute a privacy leakage risk and need to be guarded against in data applications. The study also points out that future research should further explore the differential impact of specific facial regions (e.g., eyes, mouth) on ideology prediction across genders and the applicability of these findings to non-politician samples.