President Donald Trump and other top aides were upset that Defense Secretary Mark Esper publicly opposed the deployment of active-duty forces to confront protesters in U.S. cities, as Trump has suggested, and viewed the Pentagon chief’s remarks as out of line.

Esper, speaking to reporters at the Pentagon on Wednesday, said the use of active-duty military forces to perform law enforcement responsibilities within the U.S. is “a matter of last resort” and that the National Guard is better-suited to the job.

On Monday, Trump threatened to send U.S. military forces to cities and states that fail to quell violence spiraling from protests over the death of a black man in police custody.

Mark Esper, U.S. secretary of defense, listens during a Senate Armed Services Committee hearing in Washington, D.C., U.S., on Wednesday, March 4, 2020. The U.S.-Taliban peace agreement has been disrupted but not shattered by small-scale attacks that aren't aimed at American and allied forces, Esper said.

“The National Guard is best suited for performing domestic support to civil authorities in these situations in support of local law enforcement,” Esper said. “The option to use active-duty forces in a law enforcement role should only be used as a matter of last resort and only in the most urgent and dire of situations. We are not in one of those situations now. I do not support invoking the Insurrection Act.”

The remarks generated pushback at the White House, where three Trump aides who asked not to be identified said the secretary should have moderated his comments to draw less of a distinction with the president.

Two of the officials said there hasn’t been discussion of using the Insurrection Act in more than a day so any imminent move by Trump to invoke the 1807 law was unlikely. Nonetheless, the aides said they don’t expect Trump to seek Esper’s departure.

Esper is meeting with the president Wednesday to discuss the situation in Afghanistan, according to a White House official. The meeting was not on the White House public schedule.

The 56-year-old defense chief has been a loyal aide since taking office last July, siding with the president over redirecting Pentagon funds to help build a border wall despite congressional opposition, and vowing that a vaccine against the coronavirus will be delivered by year-end with the Pentagon’s help.

Yet the nationwide protests have joined the virus pandemic as critical issues confronting the president ahead of November’s election. Trump has repeatedly called for a hard-line approach to the unrest the past week, lambasting governors on a call Monday to “dominate” their streets.

The defense chief also appeared to draw another line with the White House, saying that while he knew he would be joining Trump to walk into Lafayette Square in front of the presidential residence on Monday, he was not aware of specific plans, including what would happen when the delegation reached St. John’s Episcopal Church.

Trump and his aides have been widely criticized by many religious leaders and Democrats, as well as some Republicans and former military officials, for the visit to the church, where the president stood briefly holding up a Bible after security forces cleared peaceful protesters from the area using pepper balls and smoke cannisters.

Esper said he thought he would be reviewing damage in the plaza and at the church, but wasn’t briefed on a specific plan since he was called back to the White House after heading toward a command center at the Department of Justice.

“I did know we were going to the church. I did not know a photo op was happening,” Esper said.

Esper and General Mark Milley, the chairman of the Joint Chiefs of Staff, have come under fierce criticism for joining Trump on the trip, saying it heightened the sense of tension in America amid protests over the killing of an unarmed black man, George Floyd, by police in Minneapolis.

In his remarks to reporters, Esper offered condolences to Floyd’s family, saying his death was a “horrible crime” and a “tragedy we’ve seen repeat itself too many times” in the U.S.

As tensions over protests nationwide have risen, the military has moved more than 1,600 active duty forces into the Washington region -- including units of the storied 82nd Airborne and 101st Airborne, but not into the U.S. capital.

By Wednesday afternoon, however, about 200 soldiers were returning to their home base, the Associated Press reported, citing senior defense officials it didn’t identify, adding that more will get sent back in the coming days if conditions allow.

Retired Navy Admiral Mike Mullen, a former head of the Joint Chiefs, wrote in The Atlantic on Tuesday that “I am deeply worried that as they execute their orders, the members of our military will be co-opted for political purposes.”

“It sickened me yesterday to see security personnel—including members of the National Guard -- forcibly and violently clear a path through Lafayette Square to accommodate the president’s visit outside St. John’s Church,” Mullen wrote.

Esper’s public remarks Wednesday won praise from a predecessor: former Defense Secretary Leon Panetta.

“I commend Secretary of Defense Esper for taking the position that he took, because military leaders almost in unison do not believe that our military ought to be used to fight our own people,” Panetta, who also led the Central Intelligence Agency, said in an interview on Bloomberg Television. “It ought to be used to fight foreign enemies.”

--With assistance from Tony Capaccio, Josh Wingrove and David Westin.