In a recent paper, AI Corporate Governance and Ben & Jerry’s Risk, we critically analyze the governance arrangements of OpenAI and Anthropic. We show that these firms share an unusual built-in conflict. Each raises billions of dollars from profit-seeking investors, and then lets self-appointed individuals override investors and decide, directly or indirectly, whether and how much profit to sacrifice to ensure the firm’s AI benefits humanity. A deep and potentially unmanageable tension is hard-wired into these firms’ corporate DNA.

Such “self-appointed mission guardians” have been used only once before, at Unilever subsidiary Ben & Jerry’s. That experiment ended in spectacular failure, with the guardians causing what we call double trouble: they both harmed investors and achieved the opposite of their mission (as they saw it). Our analysis highlights the risk to firms and their investors of installing such guardians and can explain why Anthropic’s designers opted to install a “kill switch” allowing a super-majority of investors to fire its guardians.

(more…)