Yep, this is pretty much how I frame it too. I mean, male/female are just biological terms used for every animal species on earth. Trans women are male. Trans men are female. This is simply a fact. If we can't admit this then we're in deep trouble.
Calling males "women" and females "men" is a linguistic hedge. But I think that's fine. We do it in the name of politeness. And I guess, it's not really more of a hedge than calling an adoptive parent a mother or father instead of an "adoptive mother/father".
The issue isn't the politeness of using a pronoun or noun that conflicts with what your eyes are telling you (although there are implications to that), it's the way some people have completely lost track of the reality underlying that politeness.
Going back to adoptive parents, I'd never continually refer to an adoptive parent as "adoptive." That would obviously be mean and invalidating. But if there's something wrong with their child, and the doctor needs to know their medical history, we all need to be able to admit, without the slightest hesitation, that the person isn't the child's biological parent.
It's not a perfect analogy, because there are other areas, like sport, where biology matters. But yeah, I think the issue is less the word "woman" or "man" and more with the idea that honesty is transphobic.