New Zealand Initiative Explores AI Tool To Redirect Extremism Risks
People who display signs of violent extremism on artificial intelligence platforms could, in future, be guided towards human support and chatbot-led deradicalisation services. A new tool under development in New Zealand aims to address this growing concern.
The initiative reflects increasing pressure on technology companies to improve safety measures. A rising number of legal challenges have accused artificial intelligence platforms of failing to prevent, and in some cases contributing to, harmful behaviour. As a result, developers are now exploring more proactive approaches to risk management.
Expanding Support Beyond Mental Health Crises
A startup currently working with major artificial intelligence firms has begun exploring how to extend its services. Previously, the company focused on directing users at risk of self-harm, domestic violence, or eating disorders towards appropriate support networks. However, the rapid rise in chatbot usage has revealed a broader range of user vulnerabilities, including engagement with extremist ideas.
The proposed system would expand this support framework to include early intervention for individuals displaying signs of radicalisation. Discussions are ongoing with an international initiative established after a major terrorist attack in New Zealand. This collaboration would involve expert guidance on extremism while the company develops a specialised intervention chatbot.
The organisation already maintains a global network of helplines, covering numerous countries. When artificial intelligence systems detect signs of distress, users are redirected to relevant human-operated services. This model has proven effective for mental health support and now forms the basis for potential expansion into extremism prevention.
Hybrid Model Combining Technology And Human Support
The planned tool would likely operate as a hybrid system. It would combine a chatbot trained specifically to respond to early indicators of extremism with referrals to real-world services. Importantly, developers emphasise that the system would rely on expert-informed inputs rather than general training data used in standard language models.
Testing is currently underway, although no timeline has been set for release. The tool may initially be used by online moderators, as well as parents and caregivers seeking to identify and address harmful behaviour.
Experts have noted that such an approach acknowledges the importance of relationships, not just content. Addressing extremism requires more than removing harmful material; it involves guiding individuals towards constructive support systems.
Challenges Around Implementation And Oversight
The effectiveness of the tool will depend heavily on follow-up mechanisms and the quality of support networks. Questions remain about whether alerts should be sent to authorities and how to manage potential risks without escalating situations.
There is also concern that excessive moderation could drive individuals towards less regulated platforms. Research suggests that when users feel shut out, they may seek alternative spaces where harmful ideas can spread unchecked.
Developers argue that maintaining open channels of communication is crucial. Individuals in distress often share sensitive thoughts more freely with artificial intelligence systems than with people. Therefore, abruptly ending such interactions could leave them without support.
The proposed solution aims to strike a balance between safety and engagement. By redirecting users rather than excluding them, the system seeks to reduce harm while preserving opportunities for intervention.
With inputs from Reuters