Is it racist to oppose travellers' sites?

Selby District council identifies the location of a gypsy traveller site and they've already received complaints. The council leader says many of the letters objecting to the sites are racist.