No, I don't think this is it at all. I think it's that there are differences between males and females and those differences make females more vulnerable to sexual assault and rape. I think it's that the current push for self ID (where a man can be legally identified as a woman simply by declaring himself one) makes it impossible to tell the difference between trans women and men who just want to be able to walk into a female changing room.

And since self ID makes no requirement for hormones or surgery, this is a legitimate concern.

I think it's very difficult to argue that women have a bunch of "special privileges" in society. There are advantages to being female, sure, and more advantages to being male I would say. But the issue isn't holding onto "privilege" it's safeguarding rights designed to provide security and privacy.