I am grateful to the contributors to this Symposium for such thoughtful pieces, as well as to the editors of the Yale Journal on Regulation for putting the Symposium together. The ALI membership is scheduled to vote on the current draft this May, so I will use this concluding post to respond to just a few of the criticisms raised.

I generally agree with Clayton Gillette—who comments appear to be about Levitin et al. rather than my study[1]—that dicta can be relevant evidence of what the law is. This is especially so with respect to dicta in a unanimous opinion of the D.C. Circuit authored by J. Skelly Wright—an example Gillette points to.[2] Dicta appearing in a district court’s ruling on a motion to dismiss can also be relevant, though it is much less probative. There are also reasons to expect such dicta to be more difficult to code than other aspects of judicial opinions, especially when it appears in an unpublished decision. In these cases, the coding rubric will make all the difference. For example, In re JetBlue Airways Corp. Privacy Litigation suggests that to succeed on their contract claim, the plaintiffs would be required to prove that they relied on the privacy policy—a requirement contrary to the Draft’s section 2 formation rules.[3] It seems that the Reporters’ coding did not take account of that statement, as they identify the case as the “dominant precedent” for comment 9. Because the ALI has not published the coding rubric used in the six quantitative caselaw studies, we do not know what dicta it might and might not capture.

I’m especially glad that Mark Hall and Ronald Wright were able to contribute to this symposium. Their 2008 article should be the starting point for anyone who wants to engage in the quantitative study of caselaw. Two thoughts on their post. First, I agree that for the purposes of pure replication, double coding is optimal. Equally important, I would say, is to calculate and share of interrater reliability, which provides an important check on whether the question posed is one that can be reliably coded using the investigator’s rubric.[4] Unfortunately, the Reporters have published their results without including coding procedures, rubrics or interrater reliability calculations. But there is also a broader point about coding judicial decisions. In order to understand the results of a given quantitative study, pure replication might not be enough. In some instances, a dose of qualitative empiricism might be a valuable supplement, and even provide more reliable results. I continue to believe that my Article illustrates just that. Quantitative caselaw studies, when done right, can be very valuable. But, to steal Calabresi and Melamed’s metaphor, they provide only one view of the cathedral.

The second point: I used the 95-percent interval level not as the gold standard, but to illustrate the difference between a ratio of thirty-five to five and a ratio of eleven to four. Size of sample and magnitude of effect matter. I therefore strongly disagree with Steven Weise’s assertion that “the exact percentage of decisions that conclude that an online contract can be formed by a clickwrap or similar process does not matter.” But I also wonder about the aptness of Hall and Wright’s comparison of quantitative caselaw studies to studies of voter preferences. Unlike judges, voters are not asked to explain their reasons, much less adhere to stare decisis. And the studies in the Draft Restatement code not judges’ preferences or rulings, but their legal reasoning. Treating judges like voters might be appropriate for some questions, such as whether there are correlations between party affiliation and case outcomes. But the conceit of the Restatements is that studying caselaw tells us more than what judges think “at the time the sampling occurred.” The premise of the Restatements is that caselaw can tell us what the common law is. Although I would not claim that a 95-percent confidence level is essential to that project, it would be odd to suggest that empiricism in a Restatement does not attempt to predict what judges will do going forward.

I was pleased to read Steven Weise’s summary of changes to comment 9. Weise is an Adviser on the Restatement of the Law of Consumer Contracts, and so has access to drafts not yet publicly available. As far as I can tell, the ALI has still not shared Council Draft No. 5 beyond the ALI membership.[5] My first footnote makes clear that the Article discusses the most recently published draft. Weise’s suggestion that drafts shared within the ALI were “published” might reflect differences between the ALI perspective and common academic norms, including the norms governing empirical scholarship—a topic I discuss in Part V of the Article. In coordination with the Journal’s editors, I will update my footnote to clarify how I am using the term.

At any rate, it is nice to see the changes to the text of comment 9. Here is an example Weise does not mention, which is also relevant to Gillette’s post. The April 2017 Draft does not contain the word “dicta.” My understanding is that Council Draft No. 5 explains that the Reporters included dicta in their studies. This is clearly a major improvement.

Weise focuses on textual details in comment 9. In undertaking this project, I was less interested comment 9 per se than in thinking about the Draft’s use of and approach to quantitative empiricism generally. Rather than vindicating the Draft, the changes Weise highlights are evidence of the need for greater transparency and more attempts at replication. If any of the changes were made in response to my work,[6] they illustrate the value of independent reading of the data and attempts to reproduce the results. Who knows what improvements might come if other scholars were given the chance to read the decisions the Draft relies on and attempt to replicate the Reporters’ results. More generally, given the negative results of attempts to replicate so far, and lack of any positive replication, the Draft’s quantitative studies should not presently be accorded the persuasive authority traditionally given Restatements.

Gregory Klass is a Professor of Law at Georgetown Law. 

This post is the final installment of a symposium on the Draft Restatement of the Law of Consumer Contracts. All of the posts in this symposium can be viewed here.

[1] The coding in my study did not distinguish between dicta and holding. See page 66 of the Article.

[2] Transatlantic Financing Corp. v. United States, 363 F.2d 312 (D.C. Cir. 1966).

[3] 379 F.Supp. 2d 299, 325 (E.D.N.Y. 2005). In fact the statement is arguably not dicta, as the court seems to have required the plaintiffs to plead reliance to survive the motion to dismiss.

[4] Mark A. Hall & Ronald F. Wright, Systematic Content Analysis of Judicial Opinions, 96 Cal. L. Rev. 63, 112-16 (2008). See also page 64 of my Article.

[5] No draft is currently available on the public portion of the ALI website. The only draft currently on HeinOnline is the April 17, 2017 Discussion Draft, earlier drafts having been removed at the request of the ALI.

[6] I emailed the Reporters an early draft of the Article in June 2017, and first posted a draft online in July 2017.