Yep, I completely agree.
I'm hestitatnt to say what is or isn't a small price to pay regarding women's toilets. If it were possible to make all toilets everywhere into self-contained cubicles, I don't think there'd be much of an issue (except for things like peeing on the seat I guess). But that's not the reality. The rules need to work for all types of bathrooms.
But the key thing is that those spaces are meant to provide women with privacy and a degree of security. It seems deeply unfair to me to give anybody but women the final say on that.