I'm not saying he had to say anything about his sex life. He could have just said "it's none of your business, nothing was done with public funds or during office hours, it's personal" and been done with it. The problem is that he chose to lie about it. That's the higher standard I'm talking about. You have every right to hold me to a higher standard in regards to keeping national security secrets even when I'm not on duty. You have ever right to hold me to a higher standard in regards to not getting into legal trouble because if I do, it costs more taxpayer dollars than if a civilian does it. You even have the right to hold me to a higher standard in terms of physical fitness because the conditions of my position and pay grade require it.
Just as we have every right to hold him to a higher standard while in office in terms of honesty. Even though we're not his constituents, lying in public to the people goes against the very purpose of his position. What he does on his private time should certainly be his private business but if he chooses to speak to the public then every word that comes out of his mouth should be the truth, even if those words are "I don't want to talk about it"