Nope, not at all. Why would my reaction be any different? I'm not offended by somebody demonstrating their own brokeness. That's how I see the use of this word as a slur.

If this guy was a physical threat, I'd react differently to the threat of course, but the words used would be irrelevant. If somebody was threatening me physically whilst complimenting me I'd still have to react, right?

The point is that words have exactly as much power as we give them. As you say, you were taught at a young age that this word was hurtful. And now you continue to believe that. But why? If the same word can be used as a term of endearment and as an attack, surely the issue is the intent and not the word. We've completely lost sight of that.