I think that very much depends on your definiftion of very few. Intersex people, who are less than 1% of cases, are still identifiably male or female in almost all cases. Biological sex is only ambiguous in around 0.018% of cases, and these people don't usually identify as trans.

When you say a "third gender", you're talking about expression and identity, not biology. These people are almost always unambiguously male or female biologically, but express their identity in ways that aren't "typical" for their sex.

So while I understand your point that people sometimes fear those they consider "other", it's not because there's something inbetween male and female. I think this is that imprecise language I was talking about.