Why Consent Is Broken for Privacy and AI

Originally posted on Substack

The following is an excerpt from my book ON PRIVACY AND TECHNOLOGY where I summarize my thinking on privacy consent:

New technologies pose significant challenges to people’s ability to consent to the collection, use, and disclosure of their personal data. Under most privacy laws, consent makes permissible a wide array of data collection and processing. Websites, devices, and software continually attempt to induce people to consent (or pretend that people have consented) to data practices that are risky, troublesome, and unexpected.

To be meaningful, consent must not be unduly manipulated or coerced. And consent must be informed: people must be able to weigh the costs and benefits of consenting. Unfortunately, most privacy consent falls far short of these goals. In fact, privacy consent could almost be called a complete fiction.

False Legitimacy

Philosopher Heidi Hurd refers to consent as a form of “moral magic,” instantly transforming something that would otherwise be illegal or immoral into something that is permissible. Consent, she aptly notes, “turns a trespass into a dinner party; a battery into a handshake; a theft into a gift; an invasion of privacy into an intimate moment; a commercial appropriation of name and likeness into a biography.”Consent provides legitimacy—the law’s equivalent of a blessing—and legitimacy bestows power.

In the realm of privacy, the law today allows dubious or even nonexistent consent to pass as valid, conferring unwarranted legitimacy on data collection, use, and disclosure. Thus, consent in privacy ends up as a form of dark magic, a malevolent sorcery that falsely legitimizes troublesome and unwanted data practices and wrongly bequeaths power to organizations to do whatever they want with people’s data.

The Mirage of Meaningful Consent

Obtaining consent in privacy law generally involves one of two approaches: (1) the notice-and-choice approach, or (2) the express-consent approach. Neither succeeds in working effectively.

The notice-and-choice approach is employed for most data collection in the United States. Organizations post a privacy notice with information about how they collect, use, and disclose personal data. Individuals are often given a choice to opt out; if that is not an option, they can simply stop doing business with the organization. But since many privacy laws hold that inaction implies consent, those who don’t opt out are assumed to have consented.

The notice-and-choice approach is a charade.Hardly anyone reads privacy notices, and inaction can’t plausibly be considered consent. The law attempts to turn nothing into something, bestowing upon organizations a fictitious “consent” that gives them the license to use data as they desire. But this ruse is little better than the hocus-pocus of a trickster.

Under the EU’s GDPR, in contrast, consent must be express—an affirmative indication of agreement, such as clicking a button or checking a box. But even this more rigorous form of consent can verge on the illusory. People are often prodded to consent at times when they are least interested in thinking about the decision. The benefits of technologies are often instantaneous, and people receive immediate gratification for consenting. Individuals’ privacy concerns, by contrast, are often vague and abstract, with uncertain consequences far in the future. Unsurprisingly, people almost always consent—but the choice is rigged.

Lack of Understanding

Many people can’t understand the consequences of consenting to the collection, use, or disclosure of their data. The real risks can’t be discerned unless people know what will happen in the future, when their data will often be combined with other data, analyzed by algorithms, and used to make inferences, predictions, and decisions. Many algorithms—especially AI algorithms—are far too complicated for the ordinary person to understand.

The Problem of Scale

Even if people could somehow learn enough to meaningfully consent to one particular instance of the collection and use of their data, there are thousands of organizations collecting and using their data, many of which are engaging in a multitude of activities, each one requiring consent. We lack the time to read so many privacy notices or to learn enough to make informed decisions. In today’s digital age, there is so much data collection, use, and disclosure that obtaining consent each time is simply impractical.

The Law’s Futile Attempt to Fix Consent

Though privacy laws have endeavored to fix the consent process by making notices more conspicuous, users still don’t read them. Though many laws mandate that notices be easy to understand, simplistic privacy notices can’t accurately describe the implications of consenting to the collection, use, and disclosure of personal data. Vapid statements such as “We care about your privacy” and “We protect your data with reasonable security measures” are meaningless. Privacy is complicated—even for experts. I’ve studied privacy for a quarter century and I still don’t know enough to make thoughtful privacy choices. To properly evaluate the risks of providing data to an organization, I would need to talk with its chief privacy officer and discuss the various technologies with the engineers. I would need to review all privacy impact assessments, algorithms, the data the algorithms are trained on, the data-transfer agreements, data-security measures, and so on. And I’d have to do this for thousands of organizations.

In most situations involving technology and personal data, consent can never truly be meaningful, and the law is making things worse by pretending that it can. Instead, the law should accept that, in almost all cases, privacy consent is unavoidably fictional.

Murky Consent

The law often treats consent as a simple binary: either people consent—granting a license to use their personal data—or they don’t consent. It’s all or nothing. But consent should instead be understood as a continuum between full consent and non-consent.Most situations involving privacy fall somewhere in the middle of the spectrum and involve an ambiguous, contingent, and troubled consent, which I refer to as “murky consent.”

Murky consent should not confer the same legitimacy as full consent. Instead of granting nearly complete power to gather and use data, murky consent should provide a limited and highly restricted license.

Rather than try to turn the fictions of consent into facts, the law should lean into the fictions and embrace the fact that most privacy consent is murky. Murky consent lacks the legitimacy of full consent, and the law should reduce the power such consent confers. When murky consent is involved, the law should impose certain rigorous duties: (1) a duty to obtain consent appropriately, (2) a duty to avoid thwarting reasonable expectations, (3) a duty of loyalty, and (4) a duty to avoid unreasonable risk.

The duty to obtain consent appropriately would add a small degree of integrity to the fiction; although even good-faith efforts to obtain consent are likely to fail, the law shouldn’t allow duplicity and manipulation. The duty to avoid thwarting reasonable expectations would aim to ensure that people will not be surprised when they learn about how their data is being used. The duty of loyalty would require that organizations place the interests of consenting individuals first—that is, ahead of their own interests. Finally, the duty to avoid unreasonable risk would guarantee that people aren’t consenting to practices that are a bad risk for them. By ensuring that people can’t consent to things that are beyond their reasonable expectations, not in their interest, or unreasonably risky, these duties would act as a backstop to consent.

Having obtained consent, an organization today can do nearly anything it wants with a person’s data, no matter how bad the consequences might be for that person. The approach of murky consent, in contrast, essentially entails that if the story of individual consent is fictional, then the law should guarantee that it ends happily ever after for individuals.