I think this is a really tricky line to draw. First, all of us adjust our behaviour to some degree to fit in. I'm sure a big muscular white guy also feels the need to soften their behaviour so as not to intimidate. Other people imagine that the world sees them in a way that isn't accurate because they're projecting their own insecurities onto the world. They then blame other people for their feeling that they have to adapt their behaviour.
As you say, privilege, as broadly unhelpful a concept as it is, applies to everybody, and applies differently depending on a range of factors. Priviliege takes the form of youth, ablebodideness, attractiveness, intelligence, family background, education, on and on. There's imply no way to look at two people, wee that one is white and another isn't, and say the white person is more privileged. It's nonsense.
So when somebody tells me they're "dismantling their white privilege" or some such word salad, I often find that it's a way of saying "I'm one of the good guys." I've yet to have a single person say it to me and be able to explain what they're talkng about in any meaningful or concrete terms.
I think that Asian commmunities in general have racist attitudes towards black people, I'm pretty widely travelled and it's certainly been my experience, but even with that, my experiences in Asia have been overwhelmingly positive. Japan is literally my favourite place on Earth. I'm all for people doing what they can to free themselves of racist attitudes of course, but let's call them that. This coded language of "whiteness" annoys me.