Yeah, exactly. The biggest problem with this debate is how all the nuance is squashed out of it. Restrooms are different to changing rooms are different to prisons are different to rape crisis centres are different to sport.
There isn't a one-size-fits-all solution to all of these.
I wrote an article over a year ago arguing that trans people should use the restroom of the gender they identify as. As you say, are segregated by gender presentation. But a communal changing room is different. So is a prison.
And athletics, most clearly of all, are segregated by sex. Not gender presentation or identity. There are lots of masculine looking female athletes, they don't get forced to compete in the male category.
As for the "my gender changes from-minute-to-minute" people, yeah, I'm afraid I don't have much time for that. 😅