Christiano wrote that using AI models to train subsequent AI systems could result in an explosion of capabilities that their creators can’t control.
Christiano will join the board’s Safety and Security Committee, led by Carnegie Mellon University professor Zico Kolter. The committee has the final say on whether OpenAI releases new models, like Astra, which was deployed last week. Kolter has not commented publicly on the recent security incidents. OpenAI has not responded to TechCrunch’s request for Kolter’s perspective on the company’s approach to safety following those incidents.
Christiano is one of the people behind reinforcement learning (RL) from human feedback, a key technique for training large language models that he developed while working at OpenAI. He left the lab in 2021, subsequently founding the Alignment Research Center to focus on how to determine if an AI model could threaten its human creators.
“We currently train our AI agents with RL to get as much reward as they can,” he wrote Wednesday. “It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility.”
According to the frontier lab’s announcement, Christiano will continue advising the government while serving in his new role as a board member, but will recuse himself from OpenAI matters and model evaluations. However, that will hardly quell widespread concerns about the AI industry’s influence over policymaking.