Hey! Glad to meet a fellow...Aphantasiac? 😁
Yeah, I haven't met a single person who can give a definition of the "feeling" of being a man or a woman. Trans, non-binary cis, nobody. And I've asked around a lot.
I think there are a few good reasons why we can't completely move to one category, mainly based around the physical differences between men and women. But I completely agree that we'd do well to move as close to this ideal as possible.