Some members of the majority-controlled Republican Congress think that House Speaker Kevin McCarthy could be ousted from his role as speaker, based on blame being directed at him for failing to produce a timely budget proposal.

According to a Thursday New York Times article, McCarthy allegedly had private conversations with those close to him in which he said he didn’t think the GOP had the capability to deliver on a budget proposal.

McCarthy reportedly called out House Majority Leader Steve Scalise (R-La.) and Budget Committee Chairman Jodey Arrington (R-Tx.) and referred to them as “ineffective” and unable to take hard positions.

Republicans such as Rep. Ralph Norman (R-S.C.) said McCarthy was using the two GOP members he called out as scapegoats for his own failures.

“The agreements made by Speaker McCarthy, among other things, is to begin the ten-year balanced budget NOW and with his initiatives & directives,” Norman said in a statement, according to the Washington Examiner. “It’s HIS responsibility to get the 218 votes to ensure our nation’s financial security JUST AS HE DID IN SECURING THE 218 votes for speaker.”

While McCarthy denied the infighting in the GOP, Arrington and Scalise released a joint statement to Axios that read: “Our nation is staring down the barrel of a debt crisis, and my budget committee colleagues and I are focused on one thing: passing a budget that will stop this reckless spending and restore fiscal sanity in Washington before it’s too late.”

Some unnamed House GOP members told the outlet that they aren’t ready to put plans together to oust McCarthy yet, but those conversations have been brewing.

“The members I’ve spoken with are just stunned by his rebuking of his budget chair, and certainly of our leadership,” one House Republican told Axios.