Absolutely. I think Sotomayor put it nicely. Affirmative action should be a "look wider, search more" system designed to find people with potential that isn't being nurtured. These people are most likely to be found in underprivileged areas.
If you have a bunch of socio-economic advantages and you perform at the same level as somebody who lacks them, there's a very reasonable argument to be made that they've outperformed you and therefore deserve the spot. That's how I think systems like the should work.