The California State Assembly and Senate both passed the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, also known as SB-1047, on Aug. 28 and 29 respectively, in the latest move in the saga of regulating AI statewide. Next, it will go to Gov. Gavin Newsom, who can either sign it into law or veto it by Sept. 30.
What is SB-1047?
Known colloquially as California’s AI Act, and monitored closely across the country for possibly setting a precedent for state rules around generative AI, SB-1047 sets out several rules for AI developers in Silicon Valley:
- Create safety and security protocols for covered AI models.
- Ensure such models could be shut down completely.
- Prevent the distribution of models capable of what the act defines as “critical harm.”
- Retain an auditor to ensure compliance with the act.
In short, the bill provides a framework that prevents generative AI models from causing large-scale damage to humanity, such as through nuclear war or bioweapons, or from causing over $500 million in losses through a cybersecurity event.
The act defines “covered models” as those using computing power greater than 10^26 integer or floating-point operations — the cost of which exceeds $100 million, during training.
Recent version of the act includes input from Anthropic
The version of the bill that passed included some changes suggested by AI maker Anthropic and accepted by primary bill author Sen. Scott Wiener, D-Calif.
Anthropic successfully asked the state to remove language from the bill, saying companies in breach of the act could see legal action from the state’s attorney general. The newest version removes the need for companies to disclose safety test results under threat of perjury. Instead, the developers will need to submit statements, which do not have the same legal weight.
Other changes include:
- The shift in wording from AI companies having to provide “reasonable assurance” of safety to “reasonable care.”
- An exception in which AI researchers who spend less than $10 million fine-tuning an open-source covered model are not considered developers of that model.
SEE: Anthropic and OpenAI have done their own digging into how generative AI creates content, including biased content.
The bill no longer calls for the creation of a Frontier Model Division, an agency to oversee the AI industry. Instead, a Board of Frontier Models focused on forward-looking safety guidance and audits will sit within the current Government Operations Agency.
While Anthropic contributed to the bill, other large organizations like Google and Meta have expressed disapproval. Andreessen Horowitz, a venture capital firm known as a16z that is behind many AI startups, has loudly opposed SB-1047.
Why is SB-1047 controversial?
Some representatives of industry and Congress say the act will restrict innovation and make it particularly difficult to work with open-source AI models. Among the bill’s critics was Hugging Face co-founder and CEO Clement Delangue, as noted by Fast Company.
OpenAI, Meta, and Google have also penned letters to California lawmakers expressing their concerns about SB-1047, emphasizing the need for a more cautious approach to avoid hindering the growth of AI technologies.
An April study by the pro-regulation think tank Artificial Intelligence Policy Institute found that most Californians voted to support the bill as it stood at the time, with 70% agreeing “future powerful AI models may be used for dangerous purposes.”
Researchers Geoffrey Hinton and Yoshua Bengio, known as the “godfathers of AI” for their pioneering work on deep learning, publicly support the bill as well. The act will “protect the public,” Bengio wrote in an op-ed in Fortune on Aug. 15.
Earlier this week, Elon Musk took to X to express that he feels the bill should be passed. He posted: “For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that is a potential risk to the public.”
Two former OpenAI employees also wrote a letter to Newsom criticizing their former employer for opposing SB-1047.
“OpenAI opposes even the extremely light-touch requirements in SB1047, most of which OpenAI claims to voluntarily commit to, raising questions about the strength of those commitments,” they wrote. OpenAI disagrees with the researchers’ “mischaracterization of [their] position,” per Business Insider
Eight of the 52 Congressional members from California signed a letter saying the act would “create unnecessary risks for California’s economy with very little public safety benefit.” They argue that it’s too early to create standardized evaluations for AI, as government agencies like NIST are still working on creating those standards.
They suggest the definition of critical harm might be misleading, saying the bill has gone astray by focusing on large-scale disasters, such as nuclear weapons, while “largely ignoring demonstrable AI risks like misinformation, discrimination, non-consensual deepfakes, environmental impacts, and workforce displacement.”
SB-1047 includes specific protection for AI company whistleblowers under the California Whistleblower Protection Act.
The act brings up the challenges of balancing regulation and innovation
“We can advance both innovation and safety; the two are not mutually exclusive,” Wiener wrote in a public statement on Aug. 15. “While the amendments do not reflect 100% of the changes requested by Anthropic — a world leader on both innovation and safety — we accepted a number of very reasonable amendments proposed, and I believe we’ve addressed the core concerns expressed by Anthropic and many others in the industry.”
He noted that Congress is “gridlocked” on AI regulation, so “California must act to get ahead of the foreseeable risks presented by rapidly advancing AI while also fostering innovation.”
