The draft Restatement of the Law of Consumer Contracts invokes six quantitative studies of judicial decisions. Each study seeks to collect all available decisions on a legal question, published and unpublished; codes those decisions for factors such as issue, outcome, procedural posture, jurisdiction, and citations; and analyzes the coded data to determine majority rules, trends, and lines of influence. This Article reports the outcome of an attempt to reproduce the numerical results of the Reporters’ study of whether courts treat business privacy policies as part of their contracts with consumers.

Comment 9 to section 1 of the draft provides that a business’s posted privacy policy can become a term in a consumer contract in accordance with the rules of the draft Restatement. Judicial application of such a rule would have significant consequences. Section 2 describes a low bar for consumer assent to standard terms, affirming inter alia clickwrap, shrinkwrap, and browsewrap forms of assent. Comment 9 therefore suggests that a business’s posted privacy policies can become part of its contract with the consumer without the consumer’s express consent to them.

The Reporters cite as support the comment a study of fifty-one decisions between 2004 and 2015. They conclude from that study that courts are seven times more likely to recognize a privacy policy as part of a consumer contract than to exclude it from the contract; that there is a clear and increasing trend toward treating privacy policies as contract terms; and that decisions adopting this position have been more influential than those disagreeing with it. The Reporters have also published the results of this study in the University of Chicago Law Review. They have neither published the data and coding from the study, nor described their search and coding methods.

The Reporters did, however, provide the author with a partially coded list of decisions used in their study of privacy policies. Using that data, I am unable to replicate the Reporters’ results. First, using generous coding criteria, I find that only fifteen of the fifty-one decisions in the Reporters’ dataset address the question they pose. Second, whereas the Reporters find that courts are seven times more likely than not to recognize that a business’s privacy policy might be part of its contract with the consumer, I find a ratio of less than three to one. This weaker result together with the smaller sample provides significantly less support for the draft comment than the Reporters describe. Third, my analysis suggests there is not a clear and increasing trend toward treating privacy policies as contract terms. Most of the change the Reporters observe occurred between 2004 and 2010. Between 2010 and 2015 the ratio of decisions coded as recognizing privacy policies as contract terms to those holding that they are not dropped somewhat. Fourth, the Reporters use of total citation counts to identify leading cases is flawed. Examination of citing cases reveals that most do not refer to the supposed leading case for a relevant legal proposition.

The Reporters’ numerical results mask deeper problems with the study. In addition to some significant coding errors, the numbers obscure the many difficult judgment calls needed to code the decisions. Most significant among these is the Reporters’ coding of cases in which the business invoked its privacy policy as a defense against a claimed non-contractual privacy violation. Although the Reporters coded these decisions as recognizing that the privacy policy could be a contract term, courts in these cases applied consent rules drawn from tort and statutory law, not contract law. In addition, the vast majority of the decisions in the dataset are from federal trial courts, with two-thirds on motions to dismiss. These decisions are not binding on other courts, and their persuasive value is very limited. And many of the decisions that allow a contract claim to survive a motion to dismiss also include judicial statements contrary to the rules in the draft Restatement. Even if a majority of the coded decisions contain some support for the proposed comment, the degree of support in those decisions is very low.

There are four takeaways. First, this study finds that the Reporters’ data regarding the judicial treatment of privacy policies do not adequately support the conclusions they draw or the proposed comment. Second, the results of this study, especially together with those of Levitin, et al., suggest that the draft Restatement’s other quantitative case law studies should not be considered definitive until their methods have been examined and their results reproduced. Third, this study’s results illustrate the importance of transparency and replication in quantitative empirical studies. This holds all the more for attempts to quantify judicial decisions, whose coding often requires difficult judgment calls about both meaning and legal effect. Finally, part of the problem in this instance might be the American Law Institute’s procedures, which are not designed for transparency or to give outside scholars an opportunity to examine or attempt to reproduce empirical results. If the ALI wishes to use quantitative case law studies in future Restatements, it should consider revising those procedures.

Gregory Klass is a Professor of Law at Georgetown Law. This post is based on an article from the latest print edition of the Yale Journal on Regulation