While I am naturally a massive supporter of animals and their rights (being against horse racing, rodeo bulls etc.), this is a pretty tough question because while it is a commonly shared and accepted belief that "humans over animals" is the way to go, we as humans are just animals too. Other animals do not consent to being ridden, despite any means of affection they show. Therefore, riding an animal such as a horse, elephant, bull, donkey puts the rider at risk. While I believe it is wrong to ride an animal for common-sense reasons, it is also wrong because they should not be "invaded". They should not be forced to do something they don't want, and when someone rides on them, the lack of consent is definitely an invasion.
Does that make sense?