XXY people are male though, they don't just “present” that way. Their bodies produce sperm, albeit in unusually low levels. They don't produce eggs or have any of the mechanisms for doing so. Intersex people are another layer in the question of sports. Caster Semenya and Dutee Chand are good examples of this. But they aren't a third sex.
But intersex people are orders of magnitude rarer than trans people and once again, are completely different. So though I agree that there's more thought to be done (I truly sympathise with Caster's situtation and, from what I know, think she should be allowed to compete against women), intersex people are still a separate case.
And yes, there are all kinds of physical advantages within the sexes. Some girls are taller than others. Some boys are stronger than others. Not every boy or girl (or let's say male or female) gets to win. The overriding point is that if males compete against females, females will (as near as dammit) *never* get to win. The top 1% of females are all tall and/or strong and athletic and they're the ones competing for the gold. But the top, let's say 20% of males are stronger and taller and more athletic than *all* of that 1%.
This is the problem.
And while you might not be concerned about the erasure of female sports, female sportswomen are. Female college athletes who have scholarships on the line, are. There are far more of them than there are trans women who want to compete in sports. You're preferencing the feelings of trans girls over the livelihoods of biological females and I'm really not sure what the rationale is for that. Don't get me wrong, I care about trans girls rights too. As I said, they have just as much right to compete as anybody else. But that doesn't mean they have the right to compete aganst biological females. That's the issue here.