On June 30, the Center for Long-Term Cybersecurity’s Artificial Intelligence Security Initiative (AISI) hosted an online workshop centered on how AI developers and regulators can address critical incident reporting and internal use risk assessment, two critical pieces for implementation of emerging AI regulations.
For example, California’s Senate Bill 53 (SB 53), a state law enacted in September 2025, requires developers of frontier AI models to develop standards-based frameworks and report on the safety of their foundation models. Among its provisions, the law — also called the Transparency in Frontier Artificial Intelligence Act (TFAIA) — requires the California Governor’s Office of Emergency Services (CalOES) to establish a mechanism through which frontier developers (or members of the public) can report on critical safety incidents and potential catastrophic risks resulting from use of their models.
Since March 2026, the AISI team has been engaging with CalOES to provide research-backed recommendations to support the agency’s SB 53 implementation tasks, with a goal to inform the process of implementing AI regulation more broadly. The recent workshop was a forum for multi-stakeholder input that brought together more than 40 experts from government, industry, and research institutions to gather insights and best practices..
Organizations participating in the workshop (in addition to CLTC, AISI, and CalOES) included The Future Society, AI Whistleblower Initiative, Institute for AI and Law, Institute for AI Policy and Strategy, and Secure AI Project. “We are thrilled to be joined by an exceptional lineup of speakers and facilitators,” said Nada Madkour, Director of AISI, in her introductory remarks.
The event began with an overview of the project by the AISI team, followed by remarks from Thomas Woodside, Co-Founder and Senior Policy Advisor at the Secure AI Project, and Commander Matthew Sage, who leads the California Cybersecurity Integration Center. Both speakers highlighted the significance of SB 53 and its relevance to similar efforts in other states.
The bulk of the workshop was structured around two breakout discussions. The first centered on evaluating suitable methods for developers (or members of the public) to report to government agencies like CalOES on critical incidents resulting from use of their frontier AI models. The second examined how developers of frontier models can conduct internal risk assessments and report on their findings. Key takeaways from these conversations are highlighted in the following sections.
Critical Incident Reporting
The discussion on critical incident reporting was moderated by Deepika Raman, Nonresident Research Fellow at CLTC. The session began with a presentation by Jean Jeptoo, Legal Fellow at the AI Whistleblower Initiative, which supports individuals working at the frontier of AI to flag risks in a safe and legal way. In her presentation on whistleblower protections, Jeptoo highlighted the importance of governments’ role in providing a protected channel to aid whistleblowers who may be aware of potential AI risks resulting from frontier models. She noted that providing an anonymous channel for whistleblowers and ensuring periodic follow-up of reports can be effective tools in improving developers’ compliance with the law.
The Importance of Voluntary and Early Reporting
Many participants in the breakout session highlighted the importance of moving beyond reporting on four critical safety incident types: (1) unauthorized access, (2) catastrophic risk, (3) loss of control, and (4) deception. There was consensus that such incident types are unlikely to generate large report volumes because real-world incidents rarely happen outside the context of evaluations, although this may change as AI capabilities continue to increase.
Therefore, voluntary reporting by frontier companies and members of the public on near misses and “unknown unknowns” would be essential for generating reporting to complement the required incident types; such reporting can help capture what almost went wrong and inform future legislation. Stakeholders also agreed that harm categories should include other common risk types across AI systems more broadly as part of voluntary reporting.
Many stakeholders raised concerns that it can be difficult to prove that a critical safety incident was caused by a specific frontier model. One example of the attribution problem is model weight theft, where stolen model weights may not cause immediate harm, but the harm that materializes later is hard to trace back to the original exfiltration. Because attribution and causation are challenging to establish, early reporting (before harm materializes) should be encouraged, instead of waiting for a causal chain (i.e., a documented sequence of events linking model behaviors to harmful outcomes).
Reporting Form Design Considerations
Some participants highlighted that defining the classification of an incident can be challenging, especially for risk categories like loss of control and deception, which require anticipating and preparing for possible future scenarios. Free-text fields can allow reporters to submit detailed reasoning and additional information necessary to explain the classification of an incident and capture any nuance in their descriptions. However, a limitation of this approach is that data submitted through these forms can be hard to aggregate and analyze at scale. A hybrid approach that includes both a qualitative explanation box along with multiple-choice options is recommended to ensure flexibility for reporting.
Stakeholders also mentioned that reporting forms will need to evolve as reports start to come in and as AI capabilities advance. Reporting on incidents involving agentic AI systems requires specialized reporting considerations designed to capture unique factors of agents. Understanding of incident types might also evolve over time as agentic systems become more complex.
Taxonomy Mapping
On the question of how companies should report if they use a different taxonomy from the one government agencies decide to recommend, there was a broad agreement that imposing a single taxonomy would not be practical since organizations have their own internal category systems. Therefore, one recommendation is to ask reporting companies to map their existing taxonomy to a framework shaped by the relevant government agency. Given the current narrow scope of SB 53’s mandatory incident categories, establishing a taxonomy and mapping would be most useful for voluntary reporting, where the volume will likely be larger and more varied.
Internal Use Risk Assessment
The breakout session on internal use risk assessment was moderated by Alexandra Jumper and Bahrad Sokhansanj from the Institute for Law & AI, an independent think tank that researches and advises on the legal challenges posed by artificial intelligence. The discussion kicked off with a presentation by Joe O’Brien and Sambov Maheshwari from the Institute for AI Policy and Strategy (IAPS), who provided context about risk reporting for developers’ internal AI model use and highlighted why internal models pose distinctive risks compared to publicly available models.
In a lightning talk, Caroline Jeanmaire from The Future Society reinforced these points by detailing specific “loss of control” scenarios for internal deployment, such as when the capabilities of internal systems accelerate at a speed faster than organizations can account for.
Reducing Disclosure Friction
On the question of how CalOES and other agencies can structure reporting template and communications with developers to get disclosures that are both comprehensive and candid, participants in the breakout session pointed that there is often a lack of visibility into summary reports submitted to the state government. In California, for example, only lawyers at frontier companies and CalOES have visibility into submitted summary reports, preventing employees who can blow the whistle or the general public from accessing them. This structure lacks the external accountability mechanisms needed to drive high-quality reporting. To address this lack of internal visibility for employees, it is recommended that state agencies conduct tabletop exercises (TTX) with companies and third-party organizations to help staff better understand what needs to be reported.
Participants also noted that there is a lack of incentive for companies to conduct comprehensive internal use risk assessments; in California, the SB 53 reporting requirement is only triggered when internal assessments are conducted. To address this, state agencies responsible should provide a set of items for companies to report on, including specific verifiable prompts that can improve the quality of reports, instead of asking companies to respond to general questions. Baseline reporting is also important to prevent developers from using their own discretion over what to report.
Relationship Between CalOES and Third-Party Organizations
Part of the session focused on the relationship between CalOES and third-party organizations that are already conducting or developing best practices for internal use risk assessments. Participants noted the benefits of establishing a legal relationship between CalOES and which organizations, which allows third parties to double-check submissions to CalOES and verify claims made by companies about model capabilities and risks.
At the same time, others pointed out that a formal relationship between the agency and third-party organizations could potentially disincentivize companies from providing information to other independent researchers. Currently, companies share information with organizations because CalOES does not have a formal relationship with any external groups. However, if CalOES establishes a legal relationship with one specific organization, companies might feel less obligated to share information with the remaining groups. Despite these tradeoffs, there was consensus that state agencies such as CalOES should encourage companies to publicly share redacted versions of their internal assessments.
Escalation and Information Sharing Pathways
Participants also discussed the role CalOES and similar agencies in other states should play in developing pathways for escalating and sharing relevant information obtained via internal use reports, subject to applicable legal constraints. SB 53, for example, authorizes CalOES to share information with the governor, the state legislature, and federal government. However, the statute does not explain when and under what criteria this should happen. One recommendation was for agencies such as CalOES to develop a risk-specific playbook in advance to identify which federal and state agencies should be looped in for each incident type — and under what constraints.
Stakeholders also highlighted some of the legal constraints of SB 53, which limits who can access the content of submitted reports. It is currently unclear whether personnel from other agencies could be tasked to CalOES to review reports, or whether that would require escalation to the Attorney General.
For questions or comments, please contact Nada Madkour, Director of the Artificial Intelligence Security Initiative, at [nada dot madkour at berkeley dot edu] or [aisi dot cltc at berkeley dot edu].
