Sufism is a branch of Islam that emphasizes mysticism and all religions have their mystical elements. These are usually the most valid expressions of the religion because they report direct experience and take a broad perspective rather than trying to hold people to a fundamentalist reading of the scripture that contradicts everything people know to be rational.
Yes, the mysticism of Sufism reduces its ability to be treated seriously alongside the less-mysterious variants of Islam, because the other branches of Islam see it as a bit outlandish and extreme. Sufism is to Islam what Scientology is to Christianity. It is so different that people do not understand it and are afraid to approach it.
I do not believe the mysticism of Sufism reduces its ability to be treated seriously alongside the less-mysterious variants of Islam. I believe the problem with Sufi's boils down to a cultural problem with tolerance and not the tenets found in an arm of the Islam tradition. Just as different groups of Christianity don't agree it is the same with Islam.
For the same reason that a child wants to open a box on the table, people are drawn in by curiosity. Sufism has that same attraction. People begin to explore sufism as an alternative to main stream Islam. Because Islam is so sectarian, returning from Sufism is very difficult. The sectarianism of Sunni versus Shi'a may also play a factor that is driving Muslim's to a new faith.
When you read through the Koran, it is quite obviously packed full of mythology and mysticism as is the case for any religion ever created. The fact that a sect breaks off and is less popular doesn't mean that it is wrong. Eventually, most sects are treated seriously and gather a lot of followers.