Actually Caster Semenaya may well be male. If we're going to have this conversation accurately, we need to stop treating "male" and "man" synonymously. Caster is a woman. She was raised as a woman, she identifies as a woman, I have no problem recognising her as a woman. But her biology doesn't care about any of that. And as we don't know exactly what her condition is, it's quite possible that she's male (in fact I've seen numerous sources say that she is).
Yes, fundamentally speaking the top 0.1% of athletes have advantages over everybody else. But it's not so simple to describe them as unfair. If the advantages Phelps or Bolt have are unfair, how do we describe an advantage that is fair? Is a strong mentality unfair? Is a mindset that allows you to push yourself harder than others with more natural talent unfair? Should the aim be to only have people who are exactly psychologically equivalent compete against each other in the hopes that all athletics competitions are eight way ties?
As I said in the article, sport will never be a level playing field. This is fine. But female sport exists so that females can compete, and the best females can be recognised for their hard work and dedication. "Fairness" in this context simply refers to females competing against other females with no external or artificial assistance. Lots of females will struggle to compete equally in this climate. But in a climate where males and females compete together, no females can compete equally.
What we're really arguing about here is whether female sport has a place. I think it does.