The only issue is that the ghostwriter themselves doesn't get the credit but they often enjoy the anonymity. If they don't get a cut in the book's profit that is wrong. It may not seem fair that an author who was barely involved in the writing gets all the credit but the ghostwriter knows what they are getting into and it often helps them to build their credentials.
People don't really buy books because they want to see the phrasing and book constructing skills of the author, even when it's a book that they bought just for a big name. They want that person's views, in a form they can enjoy reading. I see no problem with hiring someone to write those views into a good book. Someone who was payed not to get credit, and willingly accepted, doesn't need their name on the book.
Ghostwriting involves taking the name of an author, when in reality, someone else writes the piece. Many famous authors have had their names attached to projects even though the person didn't write the actual stories. L. Frank Baum's "Wizard of Oz" series is one example. He wrote a few original stories, but not all of the published works. If there is a demand for similar stories attached to an author's name, then ghostwriting is no big deal whatsoever. Writing is for entertainment value and information dissemination in the first place. Who writes it truly doesn't matter in the long run.
The very idea behind the practice of Ghost Writing is deceitful, at best. To me, a good comparison would be a restaurant that is world famous for its Homemade crab cakes. People come from all over the nation to sample what some people refer to as "the best crab cake in the world". Now imagine that you made the effort to go to this restaurant, order their world-famous crab cake, and are instead served a frozen crab cake from Wal-Mart. And even worse, the restaurant lies to you and tells you that they made it themselves. That is just wrong, in my opinion.
The practice of ghost writing where a write will take the name of another author and write a piece on their behalf is essentially cheating your customer out an original product.
The consuming public will go to a book store to by the latest work by an author, and end up with a book that is not of the same quality as would be by the original author. This can be related to going to a big name concert, paying full price for a ticket, only to get there and realize its a cover band playing all the big name band's songs. The songs will sound similar, but its not same as seeing the original big name band.
This practice also equates to false advertising. This written work is being advertised as the latest and greatest novel by author X but it really is written by someone else.
In the end the consuming public is being cheated by not receiving and original work by the author they were expecting.