Great analogy. As you say, its not water-tight, but it makes the point.
If whiteness could even be defined in a consistent way, and one that only ever described the behaviour of white people, maybe there'd be a conversation to be had about the term's validity. But 100% of the time it seems to be a way of saying "white people bad" without having to say the words.
As you say, inevitably this validates the people with ideas about what black people are. It's a stupid gane that we all need to stop playing.