Yeah, this is exactly the mistake that leads to the concept of "nonbinary" and the idea of "feeling like a man."

As you say, somewhere along the line we bought into the fiction of a "real man/woman," and this is the "man/woman" that is at the either extreme of the "gender binary". But once we acknowledge that this "real man/woman" doesn't actually exist, the concept of the gender "binary" and the idea that there's such a thing as "feeling like a man/woman" also falls apart.

I think we should simply be arguing for the continuation of the work society (by which I mainly mean feminists) has been doing to divorce the categories of man/woman from stereotypical expectations and remove the societal pressure for men and women to conform to those stereotypes. We've made a lot of progress on that front. Man/woman are, and should remain, the terms for an adult male/female human. Like stallion and mare for horses or hen and rooster for chickens. Nothing more controversial than that.

As with those animals, there are some population level differences in behaviour between men and women. But at the individual level, there are women who are far more masculine than I am and men who are far more feminine, and this is totally fine.