Whenever you like! The reason the question isn't treated with the same urgency is that nobody is redefining the word "man" to include a class of people who present a threat to their safety and privacy.

If somebody ever tries to claim that grizzly bears are men, and tries to put them into formerly male-only spaces, you can bet that we'll be talking about it. Until then, it seems more sensible to focus on women, no?