he's somehow socially respectable, despite the aforementioned pervertry. Noone looks at him and says "look at that creep" or is like "keep your children away from littlefinger"
People trust Varys less than him, and he's completely asexual.
normal in the GoT world. Thats a very different normal from the real world.
He's a downright saint compared to Ramsey Bolton