Yes, political sex scandals should have significance, because they show the characters of those involved. A person who wants to represent the public should be a person of high character. If they cannot tell the truth about their personal life, they cannot tell the truth about how they will represent the public.
Sex scandals might be morally offensive, but it is in the private sphere. Politics is a professional field and should near absolutely be about the third person arguments, not the first, so long as the scandal does not spill over into policy. It's policies, not personal lives that must differentiate. The public tasks politicians with creating a stable economy and world stage.
The disgrace is in politicians not addressing those things, not sex scandals that are subjective, legal and potentially not offending to less religious voters. Doing the individual voter's bidding on those subjects is the proper goal of the individual voter. For some people, it would be better for them to face candidates with a brown paper bag over them, to ask them to think of the issues.
The real immorality is in not following the Constitution and our letting politicians get away with it, not sex scandals.
It shouldn't matter to anyone what happens outside of a politicians political career, especially not what happens in his or her bedroom. If a person were to have an extramarital affair, that doesn't mean they are any less capable of leading and ruling. Everyone makes mistakes and these types of mistakes are between spouses, not the entire country.