A scandal on a personal level can affect the outcome of a vote in an instant. A voters personal opinion on the moral standing of a candidate is key to getting or losing votes. If someone thinks the candidate is a terrible person for doing something, they are likely not to vote for them.
They should matter to a degree when voting, I believe in this country they have far more significance than they deserve though. If a candidate has an affair, does that question their character? Sure, and it should be considered when voting. For a lot of people that by itself will make them turn on a candidate though, which is a little ridiculous.
The American Political System should not be this way, but sadly there is a lot of soap opera non-sense factoring into today's Voting System. Campaigns are run on the concept of muck-raking. Commercials are everywhere showing gossip, rumors, and even lies that have been dug up about candidates running for office. Voters love this drama and consider it when voting.
People listen more to scandals than to the promises of the politicians. All kinds of strange reasons. Some people have refused to vote for a candidate due to supposed religious beliefs, or who they have or have not dated. Sometimes it is clear that the best man for the job could fix everything, but no one will vote for him due to some supposed Scandal.
Voters absolutely care about personal scandals because they portray who a person really is. No body wants a liar and a loser to be put in a government position. There is no way that people do not consider these because we all care about what people do in their spare time.