Yes, Vernon was abusive with Harry, although treated Dudley like any normal kid. Vernon is just a Dad who doesn't understand magic and punishes Harry a bit harshly; he's not necessarily beating the kid on a regular basis, and seems more ignorant than anything.
Then you have Petunia, Harry's blood relative. She spoils Dudley, and treats Harry like crap without exception. Not only does she treat Harry like crap, but then you gotta realize that she also withheld information from him. How much do you have to hate your nephew if you choose not to tell him anything about his own mother, your sister? Petunia hates Harry because magic is what made Lily special, while Petunia was just Petunia. She hates magic, and everything associated with the magic world, which is why she was so content with her normal life and spoils her son. So she resents Harry because she knows how special he is.
So Petunia knows all about the magic world, and Harry's parents, and keeps this knowledge from Harry, probably causing him some deep seeded psychological issues. And on top of that, she could easily have explained these things to Vernon, but instead supported every instance of punishing Harry.
It's a tough call. I actually take back what I said about Petunia. She was a b***h. The only reason I don't curse her out completely is cause unfortunately Harry is related to her. Vernon is an a--hole. So it's tough to say. I'd say the edge goes to Vernon for being abusive physically. But Petunia is responsible equally for supporting Vernon actions.