Yep, really. "Cracker" was used long before slavery in America, mostly between white people as you say, but became popularised in America in the North to refer to Southerners (cracker in this context comes from whip cracker), and as freed and escaped black slaves made their way up North, the word became a slur used by black people too.
So while you're right that it has a long history, the only reason most of us have heard of it today is because it refers to Southern slave owners and later white people in general (mainly by black people).
Also, I think the history of the n-word is a little more charged than "cracker". Yes, it comes from the Spanish for black "negro", due to Spanish colonialism. And was used as a way of differentiating black people from Europeans to justify treating them as sub-human.