You make a good point here and I couldn't agree more with the spirit of it. We are NOT our skin colour. And nobody should have to pay a price for it.
The difference is that your family came to America by choice, understanding the situation in the country and the challenges you'd face, but also recognising the opportunities you'd be given.
Part of what you inherit with that choice is a country with an ugly, fairly recent history, which it's still grappling with.
Africans were brought to America against their will, sold into slavery, and even when slavery was over, were denied even the most basic opportunities until 60 years ago. Your family's decision to come to America doesn't override that. Do you see where I'm coming from?