But there are though. There are a few areas.
Firefighting isn't one of them. As you say, if a woman wants to be a firefighter and can pass the relevant fitness tests, which many women can and many men can't, there is no reason why she shouldn't be a firefighter. I've never said anything even approaching "sex is relevant to being a firefighter," so you're arguing with your imagination there.
But sex is relevant to something like sexual assault for example. Males are, a) more likely to be sexually aggressive than females. Males commit something like 94% of all sexual violence. They're b) on average, stronger than females, making it easy to commit said sexual violence. And c) have the anatomy that allows them to commit sexual violence against the will of their victim in a way that females do not. That's three distinct factors, all localised in one sex, that directly endangers the other.
This isn't to say, of course, that all males are a danger to females (although studies show that sexual aggression amongst trans women and cis men is similar (https://fairplayforwomen.com/criminality/). Prison statistics bear that out too (https://www.bbc.co.uk/news/uk-42221629.amp)). But it's to say that this is the basis for sex segregation in instances where females will be undressed or vulnerable, and to claim that sex isn't at least relevant here is nonsensical. These are sex differences. Not gender differences.
Your argument regarding Caster is wrong on a number of levels, partly because you're overlooking the fact that Caster may well be a male with androgen resistance (nobody actually knows what her situation is because her medical results weren't released). And partly because you seem to think that testosterone (or specifically testosterone levels) is the determining factor for athletic performance. This reply is already getting a little long, so if you want to understand what you're missing, I refer you to this article I wrote about it.
https://steveqj.medium.com/the-endangered-art-of-running-like-a-girl-83c636cffdda,
