Yes, privatization is a good way to reduce government costs, because there are fewer costs involved with contracting something out. When something is hired out to private contractors, often they do not have to pay health benefits or other costs. Contract work is also easier to take away if the private company does not do a good job.
Even though there are some circumstances in which it could mean running things more efficiently, privatizing aspects of the government is a slippery slope to everything being commercialized. For example, running prisons could become a business, which means that there would be incentives to have more prisoners or different types of prisoners.