I'm well aware of that. But to argue that there's no connection between the outcomes in one generation and the outcomes in the next is naive. The three factors that you mention are all perfectly true and valid. All of them need to be addressed and affirmative action and diversity will barely scratch the surface of addressing them.
But think a little deeper and ask why certain attitudes towards education exist? Ask whether it's because children's parents grew up in an environment where the odds were heavily stacked against their success. Ask whether those atitudes will trickle down to the next generation.
Or ask how it impacts a child's attitude to higher education when they know that their parents can't afford to send them to college because of their lack of any generational wealth. Now imagine that kid knows that they can still apply for a scholarship if they work hard? If they know that there are opportunities out there that allow them to succeed.
Your argument is based solely on the "bootstraps" principle and seems determinted to ignore the many factors that lead to success. I'm a firm believer in taking responsibility for your own destiny, but I also recognise that "If I can do it why can't you?" is a simplistic way of looking at disparate outcomes.