Yeah, I think this is the problem. This is true, but it doesn't feel validating. It's the conflict between being polite and being factual.
In almost all cases, we can stick with the former. If I meet somebody in the street, if I work with them, if I interact with thin any one of a thousand ways, their sex or gender or sexuality or and number of other things make no difference. We're just two people, let's treat each other as such.
But there are a few occasions, particularly when we're talking about putting males with females, where the facts really do matter. Males outperform females by anywhere from 5.5-36.8% in sport (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3761733/). Those are enormous differences at the very top of the elite level. So sex matters.
Males commit something like 94% of all sexual assault. So even though there are many, many males who pose no threat to females, we give females sex-segrgeated spaces on occasions where they're going to be undressed or particularly vulnerable.
If we could have this conversation with people acknowledging the need for nuance, I think it would be a non-issue.