Anyone who has done research on the U.S. Federal Reserve System and who has ventured into an old-fashioned library will notice something remarkable: the shelves in the HG 2500 section of the library veritably groan under the weight of conspiratorial tomes. These range from the absurd (about international cabals that control the Fed and the nation through secret organizations like the Bohemian Grove or the Trilateral Commission) to the detestable (there is no shortage of anti-Semitism in these books).

One slice of criticism is different from the truly conspiratorial. And that criticism is the idea that the bankers “own” the central bank. Not that bankers have too much influence on Congress. Not that bankers always get their way in regulatory battles. But that the banks literally own the Federal Reserve. And since that’s true (goes the argument), then all central banking fruit comes from this poisoned tree. Why would the banker-owners of the Fed ever allow it to make policy harmful to the banks, even if in the public interest? Why should the public ever treat the Fed as anything other than a puppet for those private interests?

This argument has gained a lot of traction among the electorate. And the problem with the bankers-own-the-Fed argument, unlike those absurd and the detestable conspiracy theories, is that it is based on more than a kernel of truth.

The problem comes in both the governance structure of the twelve regional Federal Reserve Banks, and in the relationship between the Reserve Banks and the private banks. As originally conceived and implemented in 1913, the private banks did have a quasi-ownership relationship to the Reserve Banks, scattered unevenly throughout the United States. The Fed needed capital to get started; the private banks provided it in the form of “stock.” Once they bought “shares” of that “stock,” the Reserve Banks paid a “dividend” and allowed the private bankers to “elect” the Reserve Banks “board of directors,” who in turn take part in appointing the Reserve Bank presidents. Using the language of corporate law, the authors of the Federal Reserve Act described a system that gave the conspiracy theorists ample basis for their argument. If conspiracies require secrecy, then the private banks’ ownership of the Reserve Banks doesn’t really qualify. The ideas are written right there in the Federal Reserve Act.

I’ve placed an exhausting number of scare quotes in the preceding paragraph to emphasize much of the argument is a linguistic exaggeration. While the initial structure at the Reserve Banks looked much more like a private corporation organized under state law, over time the Reserve Banks looked very different. The private banks don’t have untrammeled control over the appointment of the Reserve Banks’ chief executive, for example; that authority is diluted by statute so that some of the bankers’ representatives don’t have a vote at all. And that dividend isn’t a discretionary one decided by the shareholders’ representatives in management. It’s statutory.

I’ve covered these issues before (see here for a more prescriptive and historical paper published by Brookings, here for a more legal academic piece in our own Yale Journal on Regulation, and here for my forthcoming book that touches on some of these issues). What’s new and noteworthy is that one of the most enduring and least defensible aspects of the Federal Reserve Act, part of the core of the argument that the banks own the Fed, has just changed without so much of a whimper. The statutory dividend, set in 1913 at 6% per year, was just changed to a floating rate tied to the federal funds rate set by the Federal Open Market Committee. This change is now law, just signed by the President after passing both houses of Congress.

This is a frankly stunning change. This dividend has stubbornly persisted without much of a plausible defense for over a century. Changing the Reserve Banks, even where argument and logic and public interest overwhelmingly favor that course of action, has proven politically nearly impossible. And yet somehow, this very change occurred without much of a whimper.

I don’t love that this change was pared with revenue for highway construction—the dividend is separate from the income the Fed derives from conducting monetary policy, but there is significant risk of turning the Fed into a congressional piggy bank. This is the very definition of monetizing the public debt. We haven’t crossed that Rubicon yet, but this move gets us closer than I would prefer.

That hesitation shouldn’t distract from the fact that this is an extraordinary change that should comfort those who think financial reform can only occur during post-crisis hysteria. This is a big change, and there was no crisis in sight. Reducing the dividend makes the central bank look and act more like the governmental institution it is, rather than the locus of conspiracy mongering that it often becomes. I salute the bipartisan majorities that approved this.

But reducing the dividend doesn’t go nearly far enough. We still have these “shares.” We still have a “dividend”—a 1.5% dividend in today’s environment isn’t nothing, after all. We still have those “boards of directors.” We still have those “shareholders.” The structure created in 1913, effectively abolished in 1935, still lives on in the public imagination because it lives on in the language of the Federal Reserve Act.

It’s time, then, to finish the job. And the banks themselves may well be on board. In light of the statutory change to the dividend, the banks are now pushing for a corresponding reduction in the money they spent buying Reserve Bank stock in the first place when they joined the system. (As determined by statute, banks have to put up 6% of their assets, half of which gets deposited at the Reserve Banks). The banks put up this money on the understanding that they would get this dividend. And so, they argue, a lower dividend should mean a lower initial commitment.

This argument is mostly a bad one. If the banks are serious about these ideas of “stock” and “dividends,” then they should be serious about the fact that dividends are set as a matter of policy. In a private corporation, management sets a dividend—just because a dividend is slashed doesn’t mean the corporation suddenly owes you money through a stock buyback. Congress is the Fed’s management, in a sense; they can change the dividend as they see fit.

I wouldn’t dismiss the banks’ arguments completely, though, because the Fed’s messy governance structure isn’t really of their design. It’s based on a failed model that lasted from 1913-1935 but with regrettable vestiges that continue to confuse and confound.

Finishing the job of making our central bank a public institution means getting out of the business of “stocks” and “shares” and “dividends” and “boards” and every other linguistically confusing reference to the idea that the relationship between the central bank and private banks is anything other than the regulator and the regulated.

We’ve already gone most of the way there. The Fed’s significant reforms in 1935 dumped much of the Reserve Banks’ autonomy within the system. The problem is that many within the Federal Reserve System think that the 1935 reforms either didn’t occur or shouldn’t be taken so literally (former Philadelphia Fed president Charles Plosser opined on this topic in today’s Wall Street Journal, for example).

To finish the job, I would propose the following:

First, give the banks back their money. The Federal Reserve Act treats them ambiguously as regulated entities and shareholders. They aren’t shareholders. They are regulated entities. Banks already pay the Fed interest on loans; they already pay for charged services. This capital buy-in is part of the perception problem.

Second, eliminate the dividend entirely. The 1.5% floating rate is an improvement on the old model, but it still treats the banks as shareholders.

Third, turn the “board of directors” into a “board of advisers” for each Federal Reserve Bank. Let a diverse group of community leaders from every corner come to advise the Fed. There is already a parallel structure like this for bankers, the Federal Advisory Council (once a big part of the Fed’s governance, now mostly a forgotten one). The government is advised by people all the time; but having a “board of directors” gives the image (and the fact) of private influence that we should not want in our public functions.

Fourth and finally, give the Board of Governors—those seven people, including Chair Janet Yellen, put in their place by the public’s representatives in the White House and Senate—the authority to choose the heads of the Federal Reserve Banks. They already have a role to play here, but we should make it absolute. No more allowing the bankers’ representatives so plain a role in electing their supervisors. It’s tempting to say that we should give the President and the Senate the same authority over these positions that they already have at the Board, but that would only create even more confusion at the 19-person FOMC. There’s no need for so many voices in the room with an independent base of public power.

Of course, conspiracy theorists will never go away. But if we can make these modest reforms, we will render the Fed a more comprehensible—and more plainly public—institution. We can then finally, once and for all, answer that perennial question: No, bankers don’t own the Fed. We all do.