I feel like that's an assumption as to what he thought.

But, in all honesty, the singer in any band is usually singled out by a lot of fans unless those fans happen to be musicians.

According to his 'Confess' book, when he "left" JP it originally intended for him to simply make a solo album...not exit the band:

"[He] had been wanting to do a solo project and had the blessing of his band members to do so. A studio executive told him he would have to technically 'resign' from Judas Priest to do so and he wrote a statement citing his interest in a solo project. The letter was leaked and taken out of context in that he was quitting the band. Due to personal challenges with conflict, he was unable to clarify what happened and it was over ten years before he reconnected with the band and rejoined."