Surely Julia Pierson breathed a sigh of relief as she went to bed last night. She hadn’t won plaudits for her performance on the Hill, but the testimony was over and she could now turn to fixing the problems at the agency responsible for protecting the President.

And then she opened the paper this morning and saw that the newspaper reported an even more damning instance of Secret Service failure: the Secret Service had allowed an armed ex-convict into the elevator with the President, failed to tell him, Pierson even buried it, and then followed up what now seems to be an obvious lie in her testimony before Congress.

Don’t be surprised if Pierson does not stay long on the job. The Secret Service is a federal agency located within the Department of Homeland Security whose director is appointed by the President (but not confirmed by the Senate). The statute makes no mention of removability, but the position is not one that requires Senate confirmation. Presumably, she serves at the President’s pleasure.

I suspect that pleasure has run through. President Obama is likely to ask for her resignation, praise her for her sterling record, and express optimism in the agency’s future. There may be political costs, to be sure. Lack of removability restrictions didn’t prevent significant political fallout following the firing of the nine U.S. Attorneys, nor the Saturday Night Massacre during the Nixon Administration, when he asked his political appointees to fire Archibald Cox as the Watergate Special Prosecutor: Attorney General Eliot Richardson and Deputy Attorney General William Ruckelshaus resigned in protest, and the fallout for the Nixon Administration was severe (Solicitor General Robert Bork eventually followed through on Nixon’s order). But Pierson and her cause are on the opposite side of the spectrum: there is universal recognition of the Service’s failings and now, if the press reports are to be believed, evidence of her lying to Congress.

I only know what I read in the papers about Pierson and the Secret Service, but the whole debacle has me thinking about the New York Fed, the other scandal brewing in the past week, the one at the heart of the financial regulatory system, a feature of the current bureaucracy that the President and his surrogates will have to answer for. Can the President drop the hammer on the New York Fed president for failing to live up to the President’s standards for his Administration?

Nope. The President is out of luck. There is simply no mechanism for him to express his displeasure at the New York Fed for its execution of the federal banking regulatory laws. The reason is because of the unique appointment and removal process that puts the New York Fed President in place.

The New York Fed president is appointed by the New York Fed’s board of directors, a board divided into three classes. Class A directors are bankers selected by the regulated banks (JP Morgan Chase, Goldman Sachs, and so forth). Class B directors are non-bankers selected by the regulated banks (these include, at present, investors like Silver Lake co-founder Glenn Hutchins and businessmen like Macy’s CEO Terry Lundgren). Class C directors are non-bankers selected by the Board of Governors in Washington, DC (currently Emily Rafferty, president of the Metropolitan Museum of Art and others from the community). Until 2010, the directors voted as a whole to select the Reserve Bank president; after Dodd-Frank, now only Class B and C directors take that vote. The President of the United States doesn’t show up in the picture at all.

The closest proximation we get is the President’s appointment of the seven members of the Board of Governors in Washington, DC. He makes those appointments, and chooses one among them as Chair (currently Janet Yellen).

But here is what President Obama must do if he doesn’t like, say, the way that the New York Fed has handled its member banks failure to have a conflict of interest policy pursuant to the Fed’s own regulations:

(1) Tell the Governors that he will deem their failure to comply with his request to fire the New York Fed president as “cause” for their own termination; mind you, this is to all seven governors. The Federal Reserve Act has nothing to say about the removability of the Fed Chair. (Note: LBJ sought an opinion on whether disagreement over policy could constitute “cause” under the Federal Reserve Act; his Attorney General said no.)

(2) The Governors would have to turn to the directors and say that they will be removed if they fail to fire the Reserve Bank president, as the President requested, and that such cause would be “forthwith communicated by writing . . . to the removed director,” as required by statute.

(3) The directors—all three classes, not just the ones who appointed the President in the first place—would then have to fire the New York Reserve Bankpresident, who is removable at the pleasure of the board. In other words, the bankers’ representatives would have to agree that the Reserve Bank president’s failure to be harder on them was sufficient cause to call for his head.

This is the state of the President’s oversight of the financial regulatory system. I’m no devotee of the removability-as-silver-bullet theory of agency independence, as favored by Chief Justice Robert’s opinion in Free Enterprise v. PCAOB. But I will say that placing this much supervisory authority in the hands of those so far removed from presidential control doesn’t strike me as good policy, leaving aside the constitutional problems. The President can fire at will Julia Pierson if he disapproves of her service. He can’t do the same to the President of the New York Fed.

This is why I think the time has come to make the connections between the President and his banking regulators much clearer. Make the heads of the Reserve Banks presidential appointments, and then, at least when we disagree with their actions, we can plausibly say: “That’s on the President.” It’s a stretch at best to say the same today.