Nah, don't listen to the idiots. There are certainly those who will describe everybody who doesn't agree with them as racist. But they're very far from being the majority. With the exception of your workplace, there's no need to be afraid of an accusation of racism if you're not racist. Be the bigger person. It infuriates them.