Absolutely this. You nailed it. Lived experience does provide a perspective that others might lack, it's absolutely not useless. But to pretend that it's the one true path to understanding, or even the best path to understanding, is intellectually indefensible.
As you say, we wouldn't do this in any other field.