"The facts will take us where ever they take us," he said.
Holder said the investigation will also examine whether IRS officials lied to Congress about the singling out of conservative groups.
While the allegations originated in the Cincinnati office, the Justice Department inquiry is based out of Washington, Holder said.
The comments came as all 45 Senate Republicans sent the White House a letter that called for the administration to "comply with all requests related to congressional inquiries without any delay" involving the controversy.
The letter called the scandal "yet another completely inexcusable attempt to chill the speech of political opponents and those who would question their government, consistent with a broader pattern of intimidation by arms of your administration to silence political dissent."
The clearly coordinated attacks were part of a GOP effort to increase pressure on the Obama administration over the controversy, one of three potential scandals that has the White House on the defensive less than four months into the president's second term.
According to the report by the agency's inspector general released Tuesday, the IRS developed and followed a faulty policy to determine whether the applicants were engaged in political activities, which would disqualify the groups from receiving tax-exempt status.
The controversial move began in early 2010 and continued for more than 18 months, the report said, declaring that "the IRS used inappropriate criteria that identified for review Tea Party and other organizations applying for tax-exempt status based upon their names or policy positions instead of indications of potential political campaign intervention."
IRS officials, according to the report, did not consult anyone beyond the agency about the development of the additional screening criteria. They believed that the criteria they came up with were a screening shortcut meant to help with the influx of applications, the report said.
The controversial actions began after the 2010 Supreme Court ruling in the Citizens United case that greatly expanded the ability of corporations, unions and other organizations to participate in election spending, though not through direct contributions to candidates or parties.
After the ruling, the number of politically oriented groups seeking tax exempt status as social welfare organizations under section 501 (c) (4) of the federal tax code increased greatly at a time when the federal government, including the IRS, was dealing with austerity measures that reduced or stagnated personnel and resources.
The IRS' top watchdog found that the criteria used to flag potential political applications resulted in substantial delays and the request of unnecessary information from the groups.
'Be On the Look Out'
Among the criteria used by IRS officials to flag applications was a "Be On the Look Out" list, which was discontinued in 2012, according to the report.
The criteria included:
-- Whether "Tea Party," "Patriots" or "9/12 Project" was referenced in the case file.
-- Whether the issues outlined in the application included government spending, government debt or taxes.
-- Whether there was advocating or lobbying to "make America a better place to live."
-- Whether a statement in the case file criticized how the country is being run.
-- Whether it advocated education about the Constitution and the Bill of Rights.
The investigation by the Treasury inspector general for tax administration was initiated after congressional complaints began to surface in the media in 2012 that the IRS was targeting conservative groups and holding up applications.
"Whether the inappropriate criterion was shorthand for all potential political cases or not, developing and using criteria that focuses on organization names and policy positions instead of the activities permitted under the Treasury regulations does not promote public confidence that tax-exempt laws are being adhered to impartially," the report said.
The IRS welcomed the report, saying that it agreed that aspects of its original approach in handling the influx of tax-exempt applications was inappropriate.
"The IRS is required by law to determine if organizations are engaging in a legally permissible level of political activity. Centralizing these cases was necessary to achieve consistent treatment," it said in a statement.