Haha, you expressed yourself perfectly. You write much better than many English native speakers!
And yes, it's so sad that the discussion has become like this. As you said, the real transphobes are able to use this division to frighten people and cause real harm. The extremists in the "progressive" side just don't seem to care about the damage they're doing to perceptions of trans people. And women are pretty much just being silenced completely.
All we can do is keep speaking honestly and compassionately. I don't think the extremists represent the majority of trans people. Just as the transphobes don't represent ordinary people.