Countering Violent Extremism: The FBI and the Department of Homeland Security Need to Develop Strategies and Objectives for Sharing Threat Information with Social Media and Gaming Companies

Key Chapter Title List

1. Background: Definition of Domestic Violent Extremism and Threat Categories
2. Roles and Responsibilities of Federal Agencies in Addressing Domestic Violent Extremism
3. Social Media and Gaming Companies and Their Legal Considerations
4. Domestic Violent Extremists' Use of Online Platforms for Radicalization, Recruitment, and Mobilization
5. Selected Companies' Reported Tools for Mitigating Content Promoting Domestic Violent Extremism
6. Impact of Company Leadership, Financial Considerations, and Evasion Efforts on Companies' Content Mitigation
7. Mechanisms for the FBI and DHS to Share and Receive Information with Social Media and Gaming Companies
8. The FBI and DHS Have Not Established Goals or Strategies for Sharing Information on Domestic Violent Extremists with Social Media and Gaming Companies
9. Conclusion
10. Recommendations for the Executive Branch
11. Agency Comments
12. Appendix I: Experts and Affiliated Organizations Participating in GAO Interviews
13. Appendix II: FBI and DHS Perspectives on Addressing Online Content Promoting Domestic Violent Extremism
14. Appendix III: Online Platforms' Efforts to Prevent and Counter Violent Extremism

Document Introduction

This report is a specialized audit report released by the U.S. Government Accountability Office in January 2024, in response to a congressional request, focusing on the issue of domestic violent extremists' use of social media and gaming platforms. The report aims to examine the current state of domestic violent extremism's exploitation of online spaces, evaluate the content governance practices of major platform companies, and review the information-sharing mechanisms between two core counterterrorism agencies—the Federal Bureau of Investigation (FBI) and the Department of Homeland Security (DHS)—and the private sector. The research background is based on the influence of online content promoting domestic violent extremism on several high-profile attacks in recent years, and the fact that the FBI and DHS have identified lone offenders and small groups primarily radicalized online as one of the most significant terrorism threats currently facing the United States.

The report first defines the operational definition of "domestic violent extremism" and lists five major domestic terrorism threat categories jointly identified by the FBI and DHS, including racially or ethnically motivated, anti-government or anti-authority motivated, animal rights or environmental motivated, abortion-related motivated, and other undefined threats. The report clarifies the FBI's role as the lead federal agency for terrorism investigations and domestic intelligence work, and the DHS's (particularly its Office of Intelligence and Analysis) core role in collecting, analyzing, and sharing information on emerging terrorism threats. Simultaneously, the report notes that private social media and gaming companies are not directly bound by the First Amendment, can autonomously manage platform content according to their terms of service, and are generally protected from liability under Section 230 of the Communications Decency Act.

Through interviews with 16 experts from research institutions, advocacy organizations, and academia, and in-depth research on five selected social media and gaming companies (including Discord, Reddit, Roblox, and two anonymous companies), the report reveals the multiple purposes for which domestic violent extremists use online platforms: widespread dissemination of information, mainstreaming extremist ideologies, and radicalizing, recruiting, and mobilizing others. Their strategies are highly targeted, such as using softened rhetoric on large public platforms to attract attention, then moving recruitment targets to more private or encrypted platforms for in-depth communication and action planning. Online games and their related platforms are used to build trust and social bonds, creating social spaces for the spread of extremist ideologies.

The report details a series of content moderation tools employed by selected companies to identify and remove content promoting domestic violent extremism, including terms of service explicitly prohibiting such content, machine learning tools for scanning violative content, user and trusted reporter reporting mechanisms, manual trust and safety team reviews, and design features intended to suppress violative content (such as "like/dislike" systems). However, company-level efforts are constrained by various factors, including leadership preferences, financial considerations (such as advertising revenue incentives to maintain user engagement), uneven resource allocation, and the evolving evasion strategies of violent extremists, such as using coded language, modifying audiovisual content, and cross-platform migration, making content moderation work a "whack-a-mole" challenge.

Although the FBI and DHS have established various mechanisms to share and receive domestic violent extremism threat information with social media and gaming companies—including participation in non-governmental organizations like the Global Internet Forum to Counter Terrorism, appointing industry liaison project managers, hosting annual private sector partner meetings, establishing local contacts through field offices, accepting and reviewing tips, and issuing joint intelligence products—the report's core finding indicates that neither agency has developed an overarching information-sharing strategy that clearly articulates how to identify and select partner companies, nor has it set specific goals and expected outcomes. The lack of strategy and goals prevents the agencies from comprehensively assessing the effectiveness of their communications with companies and makes it difficult to ensure that resource investments align with the overall counterterrorism mission.

Based on the above findings, the U.S. Government Accountability Office concludes that establishing program goals and strategies is crucial to ensuring that information-sharing mechanisms serve the agencies' core missions. Therefore, the report makes one recommendation each to the Director of the FBI and the Under Secretary for Intelligence and Analysis at DHS, requesting that they develop strategies and goals for sharing information related to domestic violent extremism with social media and gaming companies. Both DHS and the Department of Justice agreed with the recommendation after reviewing the draft report, with DHS's Office of Intelligence and Analysis planning to complete the relevant strategy by June 2024. This report provides policymakers, law enforcement agencies, the intelligence community, platform companies, and researchers in related fields with an authoritative assessment and actionable guidance on the online violent extremism threat ecosystem and the current state and challenges of public-private collaboration.