Slavery defaces human rights. Property rights are key to capitalism, and the most important property right is that every human has a right to own himself or herself. Slavery takes away the right of a man to own himself and his personal freedom, so it goes against the very nature of free markets. An interesting article on the topic: http://mises.ca/posts/blog/slavery-could-not-last-in-an-otherwise-free-market/
The governments were the ones who had slaves and plantations. Just like it was countries that conquered and enslaved people (like Britain and Denmark did), it was mainly a government activity. Plantations belonged to the state. That went against everything that capitalism and free market believes in.
Considering big government Dems and Republicans were the ones holding slaves to achieve their goals.