A Safety by Design Governance Approach to Addressing AI-Facilitated Online Harms

Kelly Forbes is the President and Executive Director of the AI Asia Pacific Institute (AIAPI), David Hua is a researcher for AIAPI, and Peter Brimble is the Board Chair and Vice President of AIAPI. AIAPI is a global nonprofit organization committed to strengthening the Asia-Pacific economies by facilitating the responsible development and adoption of artificial intelligence. AIAPI is an independent non-profit whose supporters include philanthropies, multilateral agencies, development partners, and government agencies.

The rapid growth of artificial intelligence (AI) is reshaping the digital landscape, amplifying existing online harms and introducing new and more complex online safety risks. Generative AI now allows individuals using consumer devices with no technical expertise to create synthetic content that is indistinguishable from authentic content. A report by Netsafe and the AI Asia Pacific Institute highlights the emerging online safety implications of AI, pointing to the rise of deepfakes as a clear example. The report details how deepfakes are used to spread falsehoods, create sexualized content, and facilitate personalized communications that target audiences with malicious messages at scale. Addressing the growing sophistication and volume of AI-facilitated online harms will require a safety by design governance approach that targets different stages of the online harm lifecycle, and considers both the distinctive technological and human dimensions of online safety.

Deepfakes are an increasingly common tool for malicious actors to inflict online harm and distort the information landscape. In the first quarter of 2025 alone, deepfake scams impersonating public figures, private individuals, and service providers to trick people into transferring money caused global losses of US$200 million. Deepfakes have also been used to create pornographic content that disproportionately targets women, youth, and children. Such material has been used to facilitate sextortion and grooming, contributing to incidents of self-harm and severe emotional distress that have prompted countries such as South Korea and Australia to criminalize the creation of non-consensual sexual deepfakes.

AI-generated content is further being used to bolster disinformation and misinformation campaigns aiming to influence public opinion and sway electoral outcomes. In Romania, concerns about foreign interference involving AI-driven disinformation led to the nullification of the 2024 presidential election. Deepfakes are also being used to promote extremism, appearing in radicalization propaganda and recruitment campaigns. One recent example is AI-generated audio of Emma Watson reading "Mein Kampf," which was used to promote Nazism.

The harmful use of deepfakes fuels growing public distrust of digital and news media, emboldening bad actors to delegitimize and discredit authentic content as being false in what is known as the “liar’s dividend.” This has been observed in online discussions concerning the Russian-Ukraine and Israel-Palestine wars, where genuine war images were questioned as AI-generated. This problem is likely to worsen as AI advances. For instance, with agentic AI on the horizon, individuals will be able to instruct AI systems to continuously generate synthetic content that adapts to real-time events (e.g., social trends and global events) without requiring repeated user input.

Given the scale and speed of these new forms of digital content creation and distribution, it would be infeasible for humans to manually moderate, but AI can be utilized to detect and remove harmful content—whether generated by humans or AI. For example, Indonesia’s Ministry of Communication and Information is using AI to help identify harmful content on social media platforms and search engines before requesting content providers to remove or block it. AI may therefore be a powerful tool for addressing the very online safety challenges it contributes to creating. However, it is not a panacea as AI struggles with context and nuance and is prone to false positives when detecting AI-generated content. As a consequence, authentic content and legitimate uses of deepfakes, such as satire or entertainment, may be wrongly flagged, for example, as cyberbullying or disinformation.

Towards a systems-level response to the online harm lifecycle

AI governance frameworks often focus on developers and deployers of AI systems, emphasizing their risk mitigation obligations. However, online harms are more often driven by the irresponsible misuse of technology and end-user behaviours. While the implementation of technical safeguards by these entities is important, they are not foolproof and can easily be circumvented (e.g., through jailbreaking efforts and unintended or emergent system behaviours). Policymakers need to be careful not to neglect the critical human element of AI-related harms — end users as perpetrators and victims — which organizational governance measures alone may fail to address.

A holistic safety by design governance approach is necessary to address the wide range of AI-enabled online safety harms through establishing interventions at different stages of the harm lifecycle. This can be framed in terms of a “Prepare-Curb-Respond” framework that combines public awareness, technical safeguards, and post-harm remedies. The “Prepare” stage focuses on reducing the susceptibility of end users to online harms through digital media literacy efforts to promote informed internet usage and awareness around the impacts of interacting with AI. The “Curb” stage calls for preventive interventions, aimed at limiting the creation and spread of harmful digital content, such as content filters and watermarking protocols that trace the origin of AI-generated content. The “Respond” stage employs reactive measures aimed at remediating the effects of harmful content, such as crisis counselling and complaint avenues to pursue legal action and content removal orders. Supporting this framework requires legal and regulatory tools that are fit for purpose to capture the scope of AI-facilitated online harms and provide adequate enforcement mechanisms.

While domestic policy interventions are crucial for addressing the local impacts of AI-enabled online harms, coordinated international governance and action are necessary to contend with their transnational reach and origin. Forums such as the Global Online Safety Regulators Network and the International Network of AI Safety Institutes can enable global collaboration at the intersection of AI and online safety. They can support knowledge sharing on effective policy responses, emerging harms, and lessons from real-world cases. Given the diversity of national conditions and the influence of global cultural, legal, and political contexts (and vice versa), international cooperation offers significant mutual benefits.

Charting a safer digital course in the AI era: An Asia-Pacific perspective

In regions undergoing major digital transformation, there is a critical window of opportunity and a late-mover advantage to apply the “Prepare-Curb-Respond” framework while AI-facilitated online harms are emerging but are not yet widespread. This is particularly relevant for the Pacific, which is in the early stages of AI adoption and beginning to make strides with its digital infrastructure. As internet access expands and AI systems grow more sophisticated, such harms are likely to become increasingly common.

Regional consensus on online safety in the Pacific is reflected in the implementation plan of the “Technology and Connectivity Pillar” of the Pacific Island Forum 2050 Strategy, which emphasizes the need for safeguards that provide culturally sensitive protection against online harms. This is particularly important for Pacific women, who face some of the highest levels of gender-based violence globally and are vulnerable to growing technology-facilitated abuse. The urgency is heightened by the region's demographics: over half of the region’s population is under 25, a group more engaged with digital media and therefore more at risk of misusing AI or becoming victims of AI misuse. For example, there has been a rising trend among youth and school students of using deepfakes to bully and prank peers, causing real harm.

Fiji’s approach to tackling AI-facilitated online safety harms offers a model for the Pacific, given its digital maturity and membership in the Global Online Safety Regulators Network. The country’s National Digital Strategy sets a goal of creating a safer digital environment through “significant reduction[s] in online scams, cyberbullying, and harmful content.” Its measures align with the “Prepare-Curb-Respond” framework.

To “prepare” the public against online harms, they have distributed over 60,000 booklets on online safety and are supporting community-based cybersecurity efforts to promote safe, informed internet use and cyber resilience. To “curb” the rapid rise of personalized AI scams, they have established a dedicated scam task force and are exploring technical measures in partnership with telecommunication providers. To “respond” against inflicted online harms, it is improving remedial avenues for victims, conducting a review of their online safety legislation to strengthen enforcement against hate and gender based abuse through content takedown processes.

Meanwhile, in Southeast Asia, where countries are further along in their digital and AI development, significant multilateral efforts are underway to address the cross-border nature of AI-enabled online harms. The Association of Southeast Asian Nations (ASEAN) has published guidance on AI governance and ethics that outlines policy recommendations for addressing the risks of generative AI, particularly in relation to deepfakes, scams, and disinformation. It has further established a regional Computer Emergency Response Team to improve coordinated regional action around combating the growing threat of cyber scams that are being amplified with AI. Most recently, Malaysia is spearheading discussions to establish an ASEAN AI Safety Network aimed at strengthening the region’s regulatory and governance capacity around responsible AI. These types of regional initiatives will be important to supporting national-level interventions for curbing and responding to online harms through greater knowledge sharing and joint incident response.

Cultivating a safer digital world will require governance frameworks that can adapt to the constant evolution of AI, particularly with the rise of agentic AI and growing public interaction and reliance on AI. This requires recognizing that technical safeguards directed at AI models are not a panacea. There are significant opportunities for innovation and leadership in measures that give more attention to the agents of misuse and victims by addressing human-centred concerns beyond model-level interventions. By aligning public awareness, technological tools, and policy measures with these considerations, AI can be harnessed more effectively to address the risks it creates and strengthen the ability of countries to combat online harms.