I think this is the problem really, "harmful" language is incredibly difficult to define. But the definition can't be "it made a few people mad".
Over the course of the special I don't think there's a single person from any background who couldn't find something to be offended by. But the point is, you're supposed to watch comedy shows looking for laughs, not for offence.
If you don't find those laughs, that's totally fair enough. But the solution is not to watch, rather than to insist that nobody else should be able to. Especially considering that, if more people actually listened, Dave has two big moments in the show where he's defending trans people. Somehow this fact has been completely lost in the noise.
There's an interesting conversation to be had about the n-word (I actually have wrtten about it - https://level.medium.com/this-is-the-way-the-n-word-dies-ab51167bf9d0). Personally, I think we'd all be much better off and happier if we stopped pretending that certain words have otherwordly power to "harm" us.