WASHINGTON (CNN) -

Sen. Rand Paul marked the 50th anniversary of the Civil Rights Act at a ceremony this week honoring the late Maurice Rabb, a renowned ophthalmologist and civil rights leader.

It was part of his aggressive outreach to African-Americans and other nontraditional GOP voters as he works to expand the Republican Party and as he crisscrosses the country laying groundwork for a potential presidential campaign.

It's a community in which he has some fences to mend.

While campaigning for the Senate four years ago, Paul sparked a firestorm for questioning parts of the historic law, especially its underpinnings that place restrictions on private property.

The law gave the federal government too much power in telling business owners what they could and could not do, he argued. While he expressed strong abhorrence for racism, he said it was the job of communities, not the government, to fix discrimination in private places by boycotting such businesses.

His argument lined up ideologically with his libertarian, limited-government leanings, but Democrats have used his comments to try to define him as a civil rights opponent.

Paul is now considered a likely presidential contender. And as the most active Republican leader in the effort to recruit African-Americans to the GOP, his comments from four years ago have become a thorn in his side.

Political strategists say the senator has gotten better at framing his arguments in a less divisive manner. But in order to lay to rest controversy over his 2010 remarks, he needs to keep doing what he's doing: Try to convince African-Americans and Democratic voters that he's an advocate for many of their views, and push his fellow Republicans to join him in the effort.

"If we're going to be the white party, we're going to be the losing party," Paul said Tuesday at the Shelbyville Rotary Club while he was in town for the event honoring Rabb.

Paul's questions about the law

In April 2010, Paul was asked by the Louisville Courier-Journal about his thoughts on the Civil Rights Act. He hailed the law for striving to end discrimination in the public domain, but he didn't fully approve of the government's role in the process.

"I think it's a bad business decision to exclude anybody from your restaurant --- but, at the same time, I do believe in private ownership," he said.

He added that he agrees any publicly-funded entities should not be allowed to discriminate, but the law shouldn't necessarily apply to private businesses. "And that's most of what I think the Civil Rights Act was about in my mind."

So how would he resolve the problem? Consistent with his small-government philosophy, he said it should be up to the people to self-correct the issue.

"In a free society, we will tolerate boorish people who have abhorrent behavior. But if we're civilized people, we publicly criticize that and don't belong to those groups, or don't associate with those people," he added.

His comments were similar to the views of his father, former Rep. Ron Paul, who famously protested a 2004 vote in the House commemorating the 40th anniversary of the Civil Rights Act. The then-congressman said the law did not improve race relations, and it diminished individual liberty.

Defending his father, Paul said the problem with the law was the unintended consequences it had on property rights.

"It's not all about race relations. It is about controlling property, ultimately," he said on CNN in January 2012.

But Patrick Maney, a history professor at Boston College, said the discussion around property rights was only a portion of the debate about the Civil Rights Act at the time of passage.

"Saying the Civil Rights Act was about property rights is sort of akin to those who say the Civil War was about states' rights," he said. "The main objection, especially among the most rabid opponents, was to racial integration, assimilation, and race mixing."

'Concerned about the ramifications'

Paul also argued that the law set a dangerous precedent that's paved a way for other restrictions on businesses, such as calorie-count regulations and smoking bans.

While visiting Howard University, a historically black college in Washington, Paul was questioned last year about his views on the law. He forcefully said he's never come "out in opposition to the Civil Rights Act or ever introduced anything to alter the Civil Rights Act."

Rather, he's been "concerned about the ramifications of certain parts of it beyond race, as they are now being applied to smoking, menus, listing calories and things on menus, and guns."

But historians say there's no direct connection between the 1964 law and the federal regulations he mentioned.

"I see absolutely nothing in the Civil Rights Act as opposed to any other law which would lead to smoking bans," said Ed Dorn, a professor and former dean of the Lyndon B. John School of Public Affairs at the University of Texas at Austin.

A more likely connection, Dorn added, would be linking smoking bans and menu restrictions to health laws mandating child vaccinations.

There is one title (out of 11) in the law that addresses private businesses. It states that certain entities are exempt from the law, such as small bed-and-breakfast type facilities, known colloquially as "Mrs. Murphy's Boarding Houses," in which there are five or fewer bedrooms and where the owner is a full-time resident.

In a May 2010 interview with MSNBC's Rachel Maddow, Paul famously suggested that if he were around in 1964, he "would have tried to modify that" section of the law and "there would have been some discussion" on property rights.

The Washington Post in 2013 ran a fact-check of Paul's comments about the Civil Rights Act over the years and gave him three out of four Pinnochios for what they described as "rewriting history."