(CNN) -- The White House wanted to have 10,000 active duty troops on the streets of Washington and other cities earlier this week to quell protesters, but Secretary of Defense Mark Esper and Chairman of the Joint of Chiefs of Staff General Mark Milley pushed back at the use of any active duty troops, according to a senior defense official.
Esper did move approximately 1,600 active duty troops to be in the Washington, DC, region to respond if needed but the approximately 5,000 National Guard troops already there never needed assistance and the active troops began to leave Thursday night.
A second defense official said Milley strongly felt the threshold -- informally described as dire circumstances -- for calling in active duty troops could not be met, opening the door to whether such a potential presidential order would be legal.
CBS first reported that the White House wanted 10,000 troops.
CNN has reached out to the White House for comment.
After violence in DC on Monday, along with a militarized law enforcement presence, Milley held phone calls over two days with President Donald Trump's arch political enemies, the Democratic leaders of Congress.
A congressional source confirmed a Milley call with Senate Democrat leader Senator Chuck Schumer on Monday. A defense official also confirmed a call with House Speaker Nancy Pelosi on Tuesday. The official said several other calls were made by Milley to Congress.
While the exact discussions with the Democratic congressional leaders have not been disclosed, by Monday night Milley had already had tense words with Trump that active duty military troops against protesters in DC was absolutely not needed despite the President's threats, several officials confirmed.