When you have both hands ready while you are driving, you can react better to traffic situations and anything that comes up. It also allows you to keep your head straight on the road. So yes, I do think that talking on the speaker phone is much more safe than handheld.
Yes, I think that talking on a speaker phone while driving is safer than using handheld cell phones, because the person at least has their hands free. With hands free, the person can react more quickly if there are problems, because they will have two hands in case of an emergency.
When you compare the use of speaker phone to that of a handheld device, I would say yes, it is safer. Some states have completely outlawed the use of phones at all while driving and I personally think this is the safest way to drive. Americans have been put in a position where we need to multi-task, but I find this unnecessary when doing something that requires as much attention as driving.
A hands free rig is certainly less distracting than holding the phone, and speakerphones are a pretty solid middle ground, so long as the phone is mounted somewhere on the dash. I use mine a lot for GPS, will they want to ban GPS because we might have to glance at the map next?