But he also insisted it would end before the tournament takes place.
At least 42 people have been killed and more than 25,000 driven from their homes in 12 days of attacks.
The mobs are accusing African migrants of taking jobs and fuelling crime.
Jordaan said the level of scrutiny of South Africa after the recent violence was an inevitable result of hosting the World Cup.
"We can see the focus of attention on our country, for completely different reasons, both of them tragic," he said.
"Our standpoint is that this World Cup must be a celebration of Africa's humanity.
"Africa has too often been a continent of division, of wars, of humiliation.
"And certainly we condemn any situation that continues to inflict on African people humiliation, suffering, war, disease.
"So our position is crystal clear and we ask that every action must be taken to stop inflicting on displaced people further displacement.
"It's something that will pass ... South Africans are not xenophobic."
Jordaan reiterated his view that all the stadiums would be ready in time for the tournament.
However he conceded that finishing a new stadium in Port Elizabeth in time for the Confederations Cup in July next year was "a challenge."
What are these?
Bookmark with:
What are these?