Yeah, I think it's almost impossible to honestly claim that CRT isn't neo-marxist. Though people waste hours trying. And when you apply a framework like Marxism to race it's impossible not to also be racist.
A singular focus on power dynamics works reasonably well when applied to workers/owners or class structures, but is far less useful when applied to human beings whose skin happens to be a different colour.
It's no coincidence that so much of CRT-infused scholarship ("whiteness" studies for example), is basically race essentialism. You can't encompass the complexity of individual human beings in a binary like oppressor/oppressed.
I agree that the Christiopher Rufos of the world have made this conversation far more difficult than it should be, and I in no way support his methods, but I'd be remiss if I didn't point out that the only reason he gained so much traction is that these crazy, racist things were already happening in schools and in the public discourse.
The racist, "all white people are trash/racist" screeds that I see so often on Medium and social media are, I would argue, a much larger influence on racial sentiment than Rufo disingenuously smearing the name of CRT. We can't normalise this kind of discourse and then be surprised that people get tired and push back. I know lots of very reasonable people who are tired of the toxicity of racial discourse. Hell, I'm tired of it.
As I said, there's no reason we can't talk about racism without attacking each other. As you pointed out, James Baldwin understood this, what, 60 years ago? The problem is that today, some people seem to have decided that the way to "progress" is to embrace racism as long as it's the "right people" being racist. That approach isn't going to end well for anybody.