This case study was written in response to this call for proposals by Holden Karnofsky to inform possible safety standards on leading AI labs. It represents ~65 hours of research, interviews, and writing. I am grateful to Michael Wara (Stanford Law School Senior Research Scholar and consultant for wildfire risk to the California Office of Energy Infrastructure Safety), Melissa Semcer (former Deputy Director of Energy Safety and Principal at Climate Wildfire and Energy Strategies, LLC), Tom Long (Director of Legal Strategy at TURN), and Marc Joseph (former shareholder and President at Adams, Broadwell, Joseph, & Cardozo and my father) to allow me to interview them for this case study.

Background and overview

This case study investigates safety standards applied to investor-owned electrical companies in California (‘utilities’) to reduce the risk of utility equipment igniting catastrophic wildfires. California investor-owned utilities are regulated monopolies responsible for purchasing and distributing electricity across the state. Three utilities cover most of the state, with Pacific Gas and Electric (PG&E) being the largest.¹

The field has many possible parallels to the current and possible future landscape for frontier AI labs. Some possible similarities are listed below (though some may become dissimilarities depending on how the AI field evolves).

• Catastrophic wildfire risk is posed by a small number of companies in California (three utilities cover most of the state).²
• It is infeasible to fully eliminate catastrophic wildfire risk (at least in short-medium term).
• The risk of catastrophic wildfires has quickly increased in the past two decades.
• During this time, risk levels and the individual sources of risk have been highly uncertain (though they are better understood now).
• The cost and effectiveness of risk mitigation strategies has also been largely unclear.
• Safety standards partially resemble a licensing scheme, with approval based on a fairly subjective evaluation across a wide range of metrics.
• Safety measures can accompany economic tradeoffs for the general public (namely higher electricity rates).
• Reputational risks play an important factor for utilities, as the largest utilities are well known by the general public and the risk posed by such companies is widely discussed in California media and politics.

This case study aims to highlight examples of company actions and regulator successes and challenges in this (potentially) similar environment, to help calibrate our expectations for AI company behavior and the risks and opportunities for safety standard strategies.

Based on my research, the following features of safety standard development on California utilities to prevent catastrophic wildfires should provide (further) caution for AI safety standard setting:

• Creating and enforcing standards while building regulator and company capacity has been a lengthy process (even after motivation among all parties was high).
• Companies and regulators did not sufficiently internalize and respond proactively to catastrophic risk as it rose (even when there were many indications that risk was growing).
• Events that look like warning shots in hindsight generally did not spark sufficient urgency around safety; only the largest and most direct disasters spurred significant regulation.
• Motives have been mixed and varied, but reputational risk and concern for public welfare have not alone been as influential as profit motive for all companies.
• Regulators were too slow to address uncertainty about the level and source of catastrophic risk (though sources of risk are much better understood now).
• It can be highly challenging for regulators to create a system with quick enough feedback loops that:
  • incorporates past utility performance into annual regulatory sign off, and
  • allows for the regulator to update guidelines on safety best practices (while evidence for the cost and effectiveness of various safety strategies is limited and changing).

This case study also explores possible benefits and challenges of several regulatory strategies that could be relevant for the AI field.

• Strict, no-fault, legal liability on utilities has been hugely important in shaping utility incentives. Still, it hasn't forced all utilities to take sufficient safety precautions.
• In the face of legal liability, penalties (or their threat) do not appear to have played an important role in motivating utility behavior, though some suggest they could be used more frequently and strongly.
• California's dependence on utilities for (affordable) electricity has limited regulator leverage to extract safety actions.
• Tying executive pay to safety outcomes can be a replicable model.
• Establishing regulation of catastrophic risk in a different regulatory body from the body that authorizes funding has created a range of possible benefits and challenges. It may have:
  • helped raise the competence of safety regulation and utility focus on safety practices,
  • contributed to limiting clarity among all parties on how utilities should best handle safety and economic tradeoffs, and
  • created bureaucratic challenges that must be navigated between various regulators.

Finally, the case study highlights:

• how private activists have seemingly played an important role in shaping safety standards, despite years of being rebuffed, and
• how voluntary safety actions by (primarily) one utility have helped inform safety policies and raised expectations for other utilities.

Contents

This case study is structured as follows:

• The problem that safety standards aim to address.
• Current standards.
• More details on each of the key findings listed above, organized by theme (e.g. proactivity vs reactivity, utility motives, etc.).

Bullet points with arrows next to them can be clicked to reveal additional detail.

The need for safety standards

Current standards

• The California Public Utilities Commission (CPUC) is the primary regulator of utilities. In 2022, the Office of Energy Infrastructure Safety (‘Energy Safety’) was created outside of the CPUC to oversee utility processes related to reducing wildfire risk.

• Core requirements for a safety certification include:
  ▷ a safety culture assessment,
an approved executive compensation structure that ties compensation to safety performance. This requirement is explained in more detail here.
an annually approved Wildfire Mitigation Plan (WMP). WMPs “describe how the electrical corporation is constructing, maintaining, and operating its electrical lines and equipment in a manner that will minimize the risk of catastrophic wildfire.” WMP evaluation also includes an assessment for a utility’s capacity to execute the plan. The text box below provides more details about the WMP components.
Demonstration of implementing its approved WMP, and board-of-director-level safety reporting structure.

Key takeaways

Proactivity vs. reactivity

• When starting from a low base, creating and enforcing standards can be a lengthy process – it took roughly 5-10+ years after it was clear that catastrophic risk was present and rising for standards to be created and have a meaningful impact on all utilities. This point is summarized in the bullets below and explored in more detail in this text box.
• Until legislation required stricter regulation (particularly starting in 2017), regulators lacked sufficient capacity and will to promptly impose rules in the face of opposition from most utilities.
• Following legislation, it took several years to build internal regulatory structure and expertise.
• For the first year or two under new regulatory structure, regulators approved safety plans with serious deficiencies on the condition that utilities would build capacity for improving safety over time.

Utility motives

• On its surface, the California utility market appears more likely than most capitalist markets to be one in which companies prioritize public welfare, including for the following reasons:
• Utilities have a highly regulated business model.
• Wildfire risk has become clear after utilities have caused deadly catastrophes.
• Utilities face significant reputational challenges. PG&E in particular has received significant negative publicity for many years related to wildfires. Its employees live among their customers, and according to one expert, employees have faced significant safety concerns while in the field due to apparent threats from members of the general public.

Activists and third parties

Regulatory structure, processes, and challenges

Influence of voluntary safety actions

Risk assessment

Audits and penalties

If you have any comments or questions about this case study, feel free to email me at coby.joseph@gmail.com.