That's because this distinction is silly at best and racist at worst. Either it matters when we judge people based on their skin or it doesn’t. Whether you call it racism or prejudice or being a doodoo-head is irrelevant.
Because you can’t partially do away with discrimination based on skin colour. You can’t condemn it in some cases and make excuses for it it in others. And given that I'm very well aware of the reality of racism, I'm perfectly willing and able to see it. I'm just not willing to trivialise it.
Acknowledging that something is “evil” is not a prerequisite to fixing it. This idea is at the root of so much of the navel-gazing, self-congratulatory ineffectiveness of modern "antiracism." Because what typically happens, is that people get all wrapped up in the fervour of their fight against evil and lose sight of the concrete issues they should be "healing."
And finally, great, if you’re not saying to consider it in all cases, which situations, specifically, should we consider it in? What criteria do we need before we ask if the president of Harvard's plagiarism and social ineptitude is the key issue?