Retrospective Review, for Tomorrow’s Sake

by Adam White — Monday, Nov. 28, 2016

In the ABA Administrative Law Section’s Report to the President-Elect, one finds a rather familiar recommendation: that the agencies undertake “careful, in-depth retrospective review of existing rules.”

I call this a “familiar” recommendation, because President-elect Trump’s predecessor called for such a retrospective review in his own Administration. In early 2011, after the mid-term elections, President Obama issued Executive Order 13563, in which he called upon his executive agencies to, inter alia, “consider how best to promote retrospective analysis of rules that may be outmoded, ineffective, insufficient, or excessively burdensome, and to modify, streamline, expand, or repeal them in accordance with what has been learned.”

And, pressing the point still further, he ordered his executive agencies to send OIRA within 120 days “a preliminary plan, consistent with law and its resources and regulatory priorities, under which the agency will periodically review its existing significant regulations to determine whether any such regulations should be modified, streamlined, expanded, or repealed so as to make the agency’s regulatory program more effective or less burdensome in achieving the regulatory objectives.”

A few months later, President Obama followed this up with Executive Order 13579, asking (if not actually “ordering”) the independent agencies to follow suit. And later that year, OIRA Administrator Cass Sunstein sent a memorandum to the heads of agencies, further detailing how the agencies should go about their work. Since then, the Administration has published reports on the status of the agencies’ efforts, although not without criticism from those casting a more skeptical eye upon the Administration’s analyses or claims.

As this experience illustrates, retrospective review is no panacea. It is challenging both in theory and in practice, and it requires the investment of more resources in the agencies to do it well. But the benefits are well worth the cost, and so I am glad that the Section included retrospective review among its recommendations.

Specifically, the Section surveys the efforts of past Administrations to undertake retrospective review, as well as ACUS’s own recommendations on the subject, and calls upon the next Administration to “institutionalize” retrospective review in a manner that would hold it to “the same standard as prospective review”—that is, the same standards that OIRA, the agencies themselves, and other authorities apply to the agencies when formulating regulations in the first place.

Proponents of retrospective review tend to think of it as a means for clearing away outdated or unduly burdensome regulations. And, true, that is among the virtues of the exercise, as ACUS’s last report on retrospective reviewshows.

But I think retrospective review’s greatest virtue actually has nothing to do with repealing regulations. Rather, retrospective review’s greatest value is forward-looking. That is, by forcing agencies to look back at their previous rulemakings and analyze their costs and benefits today, the Administration would force agencies and the public to confront how accurate or inaccurate the agencies’ own projections were in forecasting the rules’ impacts in the first place.

As we know all too well, agencies’ forecasts of costs and benefits are woefully inaccurate. As former OIRA Administrator Susan Dudley colorfully notes in her article on the subject, agencies too often “perpetuate puffery” by exaggerating rules’ benefits and understating their costs. She’s not alone in making these claims. In testimony this year before the Senate Committee on Homeland Security and Government Reform’s Subcommittee on Regulatory Affairs and Federal Management, I summarized several other reports—from ACUS to the CFTC’s Inspector General—criticizing agencies for haphazard analysis. And as Resources for the Future’s scholars observed a few years ago, independent agencies’ cost-benefit analyses are especially questionable.

Whatever the reason for the underwhelming quality of agencies’ own predictive analyses, retrospective review offers a useful antidote. By forcing agencies to go back and review their own work, under the public’s watchful eye, agencies may become more modest in their own predictions going forward. Once agencies are made to grapple seriously with the ways in which their rules’ actual impacts resemble or depart from the agencies’ own original predictions, those agencies should demonstrate great “epistemic modesty” in making predictions next time.

This is one of the major lessons to be found in Superforecasters, the 2015 book by Philip Tetlock and Dan Gardner, following decades of close study of forecasters. Reflecting on the experience of the national intelligence agencies, Tetlock and Gardner urge that forecasters should keep score of their predictive successes and failures, and that they should be held meaningfully accountable—and “meaningful accountability requires more than getting upset when something goes awry. It requires systemic tracking of accuracy . . .” (emphasis in original). At the end of their book, they tell aspiring “superforecasters” to look back at their own past errors (though without hindsight bias): “Don’t try to justify or excuse your failures. Own them! Conduct unflinching postmortems: Where exactly did I go wrong? And remember that although the more common error is to learn too little from failure and overlook flaws in your basic assumptions, it is also possible to learn too much (you may have been basically on the right track but made a minor technical mistake that had big ramifications). Also don’t forget to do postmortems on your successes too.”

This is advice that agencies need as much as anyone. Agencies are in the prediction business. The public interest depends upon the agencies becoming as accurate as possible in making those predictions. Retrospective review—institutionalized, rigorous retrospective review—is an indispensible step toward that goal.

Adam J. White is a Research Fellow at the Hoover Institution for War, Revolution, and Peace, and he serves on the ABA Administrative Law Section’s Council. Follow him on Twitter: @adamjwhitedc.


This post is part of the Symposium on the ABA AdLaw Section’s 2016 Report to the President-Elect. An introduction to the symposium is here, and all of the posts are collected here. The views in this post, which expand upon the recommendations set forth in the Report, are the author’s own and do not necessarily reflect the views of the ABA AdLaw Section. The full Report is available here.

Leave a Reply

Your email address will not be published. Required fields are marked *