No, I can think of numerous other times when it's a problem. I heard of one just a few days ago in fact. A 12-year-old girl goes into a bathroom and a trans women, who was obviously male, went in after her. She was terrified, and pleaded with a woman who happened to be in there to wait until she got out. Is this not a problem simply because the girl wan't sexually harassed?
The whole isse is simply about a sensible, not easily spoofable standard for what qualifies you as a woman. When that existed (aka when transsexuals were a thing instead of "trans" people), inclusion wasn't really an issue.A man couldn't simply "become" a woman by saying so.
As I've said to quite a few other people, bathrooms and changing rooms and prisons and rape crisis centres are all different spaces with different expectations of privacy. I've only ever heard men describe this as a simple issue. Mainly because they're not really thinking carefully about the fact that women feel vulnerbale around men in certain circumstances in a way that men never really experience.