If they're easily identified and everybody in the competition has access to them, I don't see why they should be disallowed in competitive swimming. We're not talking about body altering drugs or anything here, it's an article of clothing that doesn't pose any long term dangers, so it should be permitted in competition as long as everybody has the option of using it.
As long as any type of clothing item is allowed to all swimmers and isn't exorbitantly expensive so that it would provide an unfair advantage, then there isn't anything wrong with allowing it in competetive sports. After all, an item of clothing is not a drug. There is nothing inherently unfair.