Moving Forward with Regulatory Lookback

President Obama has rightly called on government agencies to establish ongoing routines for reviewing existing regulations to determine if they need modification or repeal. Over the last two years, the White House Office of Information and Regulatory Affairs (OIRA) has overseen a signature regulatory “lookback” initiative that has prompted dozens of federal agencies to review hundreds of regulations. This regulatory initiative represents a good first step toward increasing the retrospective review of regulation, but by itself will do little to build a lasting culture of serious regulatory evaluation. After all, past administrations have made similar review efforts, but these ad hoc exercises have never taken root. If President Obama is serious about institutionalizing the practice of retrospective review, his Administration will need to take further steps in the coming years. This essay offers three feasible actions – guidelines, plans, and prompts – that President Obama’s next OIRA Administrator should take to move forward with regulatory lookback and improve both the regularity and rigor of regulatory evaluation.

Responding to an executive order from President Obama, dozens of federal agencies over the last two years have undertaken extensive reviews of the regulations on their books, looking for antiquated, counterproductive, or unnecessary rules that should be modified or eliminated. According to the Administration, agencies have collectively completed more than five hundred regulatory reviews and initiated policy modifications expected to yield cost savings in the billions of dollars. These results look good, to be sure, but they are only a small step toward achieving the Administration’s broader goal of institutionalizing retrospective regulatory analysis. To avoid squandering the progress made so far, the Administration must use the next several years to take additional steps to improve retrospective regulatory analysis and identify still better targets for the application of more rigorous evaluation research.

The Obama Administration has sometimes characterized its existing retrospective review initiative as “historic”¹1.Press Release, White House Office of the Press Sec’y, White House Announces New Steps to Cut Red Tape, Eliminate Unnecessary Regulations (May 10, 2012), http://www.whitehouse.gov/the-press-office/2012/05/10/white-house-announces-new-steps-cut-red-tape-eliminate-unnecessary-regul. and “unprecedented.”²2.Cass Sunstein, A Smarter Approach to Regulation (August 7, 2012), http://www.whitehouse.gov/blog/2012/08/07/smarter-approach-regulation. But actually it is far from unprecedented. President Clinton issued an executive order requiring agencies to develop programs by which they would “periodically review” existing regulations,³3.Exec. Order No. 12,866, 58 Fed. Reg. 51,735 §5 (1993). and Vice President Gore oversaw a government-wide regulatory review process that trimmed a sizeable number of pages of outmoded rules from the .⁴4.See, e.g., John Kamensky, Assistant to the Deputy Dir. of Mgmt., U.S. Office of Mgmt. and Budget, The U.S. Reform Experience: The National Performance Review, Presentation at Indiana University at the Conference on Civil Service Systems in Comparative Perspectives, Indiana University (April 6, 1997), http://gov-info.library.unt.edu/npr/library/papers/bkgrd/kamensky.html. Under President George W. Bush, the White House Office of Information and Regulatory Affairs (OIRA) invited members of the public to nominate existing rules needing review and reconsideration, a process which led to the scrutiny of nearly four hundred rules and regulatory guidance documents.⁵5.Office Of Mgmt. & Budget, Stimulating Smarter Regulation: 2002 Report To Congress On The Costs And Benefits Of Regulations And Unfunded Mandates On State, Local, And Tribal Entities 4 (noting that 316 regulations and guidance documents were considered in 2002, in addition to 71 in 2001) http://www.whitehouse.gov/sites/default/files/omb/assets/omb-/inforeg/2002_report_to_congress.pdf.

Although retrospectively reviewing regulation is far from new, what makes the Obama Administration’s latest round of review distinctive is its laudable but ambitious goal of institutionalizing the practice of what the Administration calls regulatory lookback.⁶6.Exec. Order No. 13,610, 77 Fed. Reg. 28,467 (May 14, 2012) (calling for agency action “to institutionalize regular assessment of significant regulations”). President Obama’s first OIRA Administrator, Cass Sunstein, proclaimed that the Administration’s lookback would not be a “one-time endeavor” as in previous administrations; instead, the Obama Administration’s lookback aspires to be just a first step toward building “a regulatory culture of regular evaluation.”⁷7.Cass Sunstein, Regulation: Looking Backward, Looking Forward, Address Before the 2012 A.B.A. Admin. L. & Reg. Pract. Sec., Washington, D.C., May 10, 2012, http://www.whitehouse.gov/sites/default/files/omb/inforeg/speeches/-regulation-looking-backward-looking-forward-05102012.pdf.

Widespread acceptance of continuous regulatory review is exactly what is needed to fulfill what President Obama has rightly characterized as the government’s duty to “measure, and seek to improve, the actual results of regulatory requirements.”⁸8.Exec. Order No. 13,563, 76 Fed. Reg. 3, 21 (2011). Unfortunately, the federal government’s treatment of retrospective regulatory review still lags far behind agencies’ practice of prospectively analyzing proposed regulations, a process institutionalized by President Reagan and overseen by OIRA for the last thirty years. It is fair to say that retrospective review is today where prospective analysis was in the 1970s: ad hoc and largely unmanaged.

Without doing more, the Obama Administration’s recent lookback initiative will end up in the same dustbin as the regulatory review processes initiated under Clinton and Bush. Sure, some discrete improvements in specific regulations will likely result, but retrospective review will remain a periodic and unsystematic fancy rather than a serious, ongoing part of regulatory policymaking.

How to move forward? One way would be to create a new, independent regulatory institution dedicated to retrospective review, along the lines of proposals offered by, among others, Michael Greenstone of MIT and Michael Mandel and Diana Carew of the Progressive Policy Institute.⁹9.Michael Greenstone, Toward a Culture of Persistent Regulatory Experimentation and Evaluation, in New Perspectives On Regulation 111 (David Moss & John Cisternino, eds., 2009); Michael Mandel & Diana G. Carew, Progressive Pol’y Inst., Regulatory Improvement Commission: A Politically Viable Approach To U.S. Regulatory Reform (2013), http://www.progres-sivepolicy.org/2013/05/regulatory-improvement-commission-a-politically-viable-approach-to-u-s-regulatory-reform/. There is much to be said for such proposals. But as anyone who follows Washington politics knows, it will undoubtedly take considerable time—not to mention clout—before Congress might enact even such appealingly bipartisan proposals. Even if a new institution were to be authorized, funded, and staffed, it would take still more time for that body to begin to conduct reviews and make recommendations. The Obama team would likely be in its closing days, if not gone from Washington altogether, by the time a new institution could begin to have an impact.

Fortunately, the Obama Administration does not need to wait for the creation of a new institution before taking steps to embed evaluation more deeply and permanently into the regulatory process. Acting entirely on its own, the Administration can still move forward with action that will help institutionalize retrospective review for the next three years and beyond. Specifically, the White House’s OIRA should issue government-wide regulatory evaluation guidelines, require the creation of evaluation plans for significant rules as part of the prospective review process, and adapt the practice developed by George W. Bush’s OIRA of issuing “prompt letters” so as to promote targeted, value-added regulatory evaluation.

Evaluation Guidelines. OIRA first needs to establish specific guidelines for agencies to follow in conducting retrospective evaluations of existing regulations. At present, far too many agencies’ reviews are little more than glances in the rearview mirror, drawing mainly on anecdotes and expert impressions. Glances back may be better than nothing, but they fall far short of what it will take to create a credible, evidence-based approach to regulation. Rather than relying on impressions, the federal government needs careful, systematic research that addresses the question of causation: What benefits and costs can actually be attributed to a regulation after it has been implemented?¹⁰10.Cary Coglianese, Evaluating the Impact of Regulation and Regulatory Policy, (OECD Expert Paper No. 1, 2012). Getting reliable answers to this causal question requires adherence to exacting standards for research design and statistical analysis, yet federal agencies currently lack clear guidance about how to conduct high quality retrospective reviews.

It is instructive that when it comes to producing prospective regulatory analysis, agencies can turn to OIRA’s Circular A-4, a lengthy document that provides both a general guide to conducting regulatory analysis as well as concrete prescriptions for analysts to follow. Circular A-4 offers regulatory analysts in agencies specific instructions, such as, “You should not use benefit transfer in estimating benefits if resources are unique or have unique attributes,” and, “You should provide estimates of net benefits using both 3 percent and 7 percent” discount rates.¹¹11.Office Of Mgmt. & Budget, Circular A-4: Regulatory Analysis (Sept. 17, 2003).

Admittedly, some of what can be found in Circular A-4 may also be helpful in conducting retrospective analysis, but nothing in A-4 offers a specific framework for approaching retrospective evaluation. The Circular contains nothing about making causal attributions by estimating counterfactuals or about how to undertake statistical analysis of regulatory impacts. If the Obama Administration is serious about deepening and strengthening regulatory review, at the very least it should create retrospective evaluation guidelines comparable to Circular A-4.

Evaluation Plans. Issuing evaluation guidelines is not only the most feasible action the Administration could take in the near term, it also would provide a foundation upon which to base additional steps. One such additional step would be to require agencies to include in each prospective regulatory impact analysis (RIA) a plan for the subsequent evaluation of the proposed rule. An evaluation plan would constitute only a small part of an overall RIA, and it would be non-binding in the sense that an agency would not be obligated to carry out the plan. Nevertheless, such a plan would provide a future guide whenever the agency, OIRA, or the public does later deem it appropriate to look back at the rule after it has been implemented. A plan for retrospective evaluation should, among other things, discuss:

• ways of operationalizing the proposed rule’s objectives, specifying metrics that could be used in the future to assess whether each objective had been met;
• sources of data that either currently exist or would need to be developed in order to estimate the impact of the rule on the specified metrics;
• the time frame when the rule’s objectives could be expected to accrue or, relatedly, the time frame when retrospective evaluation would be appropriate; and
• research or analytic designs that could be used in evaluating the rule (e.g., sources of cross- sectional or longitudinal variation, other potential explanatory factors that might need to be controlled, and possible statistical approaches to estimating counterfactuals).

An evaluation plan would be useful if the agency later re-examined the rule in a future administration’s lookback process. Such planning also would help prompt agencies early in the rule development process—even when proposed rules are being drafted—to begin to think about retrospective evaluation needs, such as what data could be collected or identified in advance of the rule’s implementation in order to facilitate subsequent measurement. Evaluation plans should be made publicly available, and as such they may help stimulate independent evaluation research by other entities, including the Administrative Conference of the United States, the National Academy of Sciences, and the National Academy of Public Administration, as well as by university and think-tank researchers.

OIRA is well positioned to oversee formal evaluation planning as part of regulatory development. Imposing a requirement for the submission of formal evaluation plans would serve to implement the periodic review of existing significant regulations demanded under both Section 6 of Executive Order 13,563¹²12.Exec. Order No. 13,563, 76 Fed. Reg. 3,821, 3,822 (2011). and Section 5 of Executive Order 12,866,¹³13.Exec. Order No. 12,866, 58 Fed. Reg. 51,735, 51,739 (1993). not to mention the provisions of Executive Order 13,610.¹⁴14.Exec. Order No. 13,610, 77 Fed. Reg. 28,467, 28,469 (2012). In addition, since the data needed for evaluation may at times call for information-collection requests, OIRA’s role in implementing the Paperwork Reduction Act¹⁵15.44 U.S.C. §§ 3501-3521 (2012). would also make it an appropriate entity to interact with agencies over plans for evaluation.

Evaluation Prompts. Finally, OIRA should extend to the context of retrospective review the earlier OIRA practice of sending agencies occasional prompt letters.¹⁶16.See John D. Graham, Saving Lives Through Administrative Law and Economics, 157 U. Pa. L. Rev. 395, 460-463 (2008) (describing the development and use of OIRA’s regulatory prompt letters). Evaluation prompts would identify specific existing rules that the Administration believes should be targeted for in-depth review, above and beyond whatever the agency may do in the ordinary course of the ongoing lookback process called for under the existing executive orders.

Far too many of the retrospective reviews that agencies have conducted to date have been impressionistic, rather than systematic or rigorously empirical. Of course, that is to be expected with the short time frame agencies have been given under the retrospective review initiatives in recent administrations. Moreover, for some rules no more than a close but informal glance back will be warranted. For other rules, though, a more in-depth, serious evaluation will be needed to advance the goals of sound regulatory governance.

As with regulatory plans, OIRA is well positioned to implement the prompt letter proposal given its familiarity with rules across the entire sweep of the federal government. Better than any other entity, OIRA can determine what the federal government’s top priorities for regulatory evaluation should be. Specifically, OIRA should issue prompt letters calling for in-depth evaluation in at least three types of cases:

• Close calls. Rules should be evaluated rigorously when they had, at the time they were promulgated, high expected costs or benefits but relatively small expected net benefits in their RIAs. If the costs of such a rule turned out after implementation to be substantially larger than estimated, or the benefits substantially smaller, the rule would no longer have benefits that justify its costs.
• High uncertainty. Relatedly, rules expected to impose high benefits or costs merit subsequent evaluation if the prospective benefit or cost estimation exhibited high levels of uncertainty. For these rules, a follow-on investigation would reduce the uncertainty.
• Common issues. Rules that present common issues of either benefit or cost estimation – or that rely on common assumptions – are prime candidates for rigorous retrospective review, as serious efforts to evaluate their benefits and costs retrospectively would help validate or improve prospective estimation techniques applicable to other rules.

Although OIRA lacks the capacity to conduct the needed rigorous retrospective evaluation research on its own, it is distinctively positioned to help identify opportunities like these, where evaluation could assist in improving regulatory outcomes, reducing regulatory burdens, or validating or improving methods of regulatory impact analysis. OIRA could, of course, also welcome other agencies or members of the public to make suggestions for rules that should be subjected to evaluation prompt letters.

OIRA’s prompt letters would urge agencies to allocate internal agency research funds to conduct in-depth empirical evaluations of rules in accord with OIRA’s evaluation guidelines. Agencies could alternatively seek assistance from entities such as the National Science Foundation or the National Academy of Sciences to fund or facilitate systematic regulatory assessments. Either way, given OIRA’s placement within the Office of Management and Budget, it may be positioned to help support the allocation of necessary budgetary resources for its priority regulatory evaluations. OIRA’s statutory role in overseeing the Paperwork Reduction Act also positions it to stand ready to process expeditiously the approvals of information requests that may be needed to collect the data needed to undertake evaluations subject to prompt letters.

* * *

These three proposals—evaluation guidelines, evaluation plans, and evaluation prompts—could all be implemented without any congressional action shortly after the confirmation of the next OIRA Administrator. Although these three steps by themselves will not cure everything that ails the nation’s regulatory system, they nevertheless represent meaningful steps toward better regulatory analysis and ultimately better regulation. Evaluation, after all, is needed to identify both real successes and real problems that need fixing.¹⁷17.See Cary Coglianese, Thinking Ahead, Looking Back: Assessing the Value of Regulatory Impact Analysis and Procedures for Its Use, 3 Korea Leg. Res. Inst. J. L. & Leg. 5, 18 (2013) (S. Kor.) (discussing how evaluation research seeks “to attribute, causally, both the good and bad outcomes to regulations”).

Institutionalizing rigorous evaluation practices will by no means come easily. Rigor and quality have not always described even the prospective regulatory impact analyses that agencies have been required to complete under OIRA’s oversight for the last several decades.¹⁸18.”See completed between 1982-1999 and concluding “that many RIAs are of poor quality”).”] With time, though, the practice of regulatory analysis can improve and deepen. Building a culture of retrospective evaluation is a long-term proposition, and at this juncture it requires taking steps to maintain the momentum the Obama Administration has generated with its extensive lookback initiative. The only way to advance the administration’s admirable objectives of improving both regulation and regulatory evaluation is to keep moving forward with looking back.

Cary Coglianese is the Edward B. Shils Professor of Law, Professor of Political Science, and Director of the Penn Program on Regulation at the University of Pennsylvania Law School. This essay is based upon remarks delivered at a Progressive Policy Institute (PPI) forum on “Regulating in the Digital Age” in Washington, D.C., on May 9, 2013. The author is grateful for helpful comments from Brady Sullivan, Jonathan Wiener, participants at the PPI forum, and the editorial team at the Yale Journal on Regulation.