The new OpenAI team would oversee and evaluate the development of “frontier AI models” to watch out for risks including nuclear war, cyberwarfare, and pandemics.

OpenAI is taking significant steps to mitigate the potential “catastrophic risks” associated with cutting-edge artificial intelligence models. This initiative, known as Preparedness, is tasked with overseeing and evaluating the development of highly capable AI models that could possess dangerous capabilities, particularly in areas like cybersecurity and nuclear threats. The team, integrated into OpenAI, will ensure that these models adhere to strict safety protocols and guidelines.

Some specific concerns include the ability of AI models to manipulate human users through language and perform autonomous tasks. Additionally, OpenAI is actively addressing “extinction-level threats” such as pandemics and nuclear warfare. (Forbes)

The leadership of Preparedness falls to Aleksander Madry, the director of MIT’s Center for Deployable Machine Learning, who has temporarily joined OpenAI.

Another key objective is the formulation and maintenance of a “Risk-Informed Development Policy.” This policy will outline OpenAI’s approach to managing risks posed by advancing AI models, especially as they approach “artificial general intelligence.”

OpenAI is also encouraging external input through a challenge, seeking ideas on how AI might be misused to cause real-world harm. This proactive approach reflects OpenAI’s commitment to responsible AI development.

The backdrop to these efforts includes a growing awareness within the AI community about the potential risks of advanced AI models. Prominent figures, including OpenAI’s CEO Sam Altman, have emphasized the importance of addressing these risks while continuing to advance AI technology.

Related Story:  In Five Years’ Time, AI Could Escape Human Control – Geoffrey Hinton, the Godfather Of AI

Photo by benjamin lehman on Unsplash