The rule nobody wrote

In the spring of 2011, a postdoc in Michael Eisen's lab went to Amazon to buy a copy of Peter Lawrence's The Making of a Fly, a standard reference for fruit-fly biologists, and found two new copies offered at more than $1.7 million each.¹ Eisen watched the page for a week and worked out what was happening. Once a day one seller set its price at 0.9983 times the other's, and the other then reset its price to 1.270589 times the first.¹ Each rule made sense alone: undercut a rival by a hair, or, as Eisen guessed of the second seller, list a book you do not hold at a margin above the price you would have to pay for it elsewhere.¹ Together they multiplied the price by about 1.27 a day. It peaked at $23,698,655.93, plus $3.99 shipping, and fell to $106.23 the day after, when, in Eisen's words, "somebody ultimately noticed."¹

Nobody set that price. More and more of what governs behaviour online is like it: an outcome that nobody designed as a whole.

In 2000 Lawrence Lessig argued that cyberspace has a regulator of its own.² He called it code, "the software and hardware that make cyberspace as it is", which "sets the terms on which life in cyberspace is experienced."² His worry was who would write the code. The regulator people live with now often has no single writer. It is a sum.

The parts of the sum are each designed with care, and many are chosen by experiment. In 2017 Ron Kohavi and Stefan Thomke reported that Microsoft, Amazon, Booking.com, Facebook and Google each run more than 10,000 online controlled experiments a year, and that at Microsoft about a third of the ideas tested work, a third do nothing and a third make things worse.³ A product shaped this way is whatever survived many local tests, each judged by its own team's measure. What the survivors do in combination is nobody's test.

In early 2021 a father photographed an infection on his toddler's groin at a nurse's request, so a doctor could see it before a video consultation, and the photos backed up to his Google account.⁴ Two days later Google locked the account for harmful content that was "a severe violation of Google's policies and might be illegal."⁴ The lock came from a deliberate policy against child sexual abuse material, which Google says includes review by child-safety experts who consult paediatricians.⁴ The same account held his email, his contacts, his photos and his phone number, since his mobile service was Google Fi, and he lost them all.⁴ The San Francisco police investigated, found that no crime had occurred, and could not reach him to say so, because the number they had was the one he had been locked out of.⁴ Google denied his appeals and stood by its decision.⁴

Every part of that chain has an owner: the classifier, the reviewers, the reporting law, the account system, the phone service. The outcome, a cleared man whom the police cannot call, has none. Probably nobody decided, as such, that a safety flag on a photo should also disconnect a telephone; it followed from tying everything to one account.

An appeal is a request to an author: reconsider the rule, or apply it properly. Against a sum there is no one to ask. Each owner can show that their part worked as designed. The person on the receiving end may be governed by a rule that nobody anywhere could state.

If that is how much of online life is now governed, accountability has to attach to the sum itself: someone whose job is the composition, who answers for what the parts do together, and who can be reached when they combine badly. The book about flies needed only one curious biologist to notice. Most people governed by a sum cannot see enough of it to notice anything.