There is so little the parties we have can agree on. A fiscally Conservative party that just stays out of the hot social issue fussing might appeal to more and more people. There is some fatigue setting in among people who just want the economy fixed and don't want to hear any more about social issues. It's not so much that people are libertarian as it is that many want the priorities of government and politicians to be on "nuts and bolts"
The first-past-the-post system in general simply doesn't allow for third parties to really gain significant enough ground to become even close to as popular as other parties. Whether you support libertarianism or not, supporting the party is not only futile, but goes against your interests by sacrificing your vote for your preferred choice.
Having a platform placing liberty in front of social welfare, law, and financial stability signals massive red flags about the competence of those running for a seat. Let's just keep voting conservative because at least we know they can run a government. Experimenting with ideology's this for in our civilization would be for the most part unpredictable.