Isis is based on irrationality and fear. When they publicly execute one of their own fighters they do it to promote that fear. Motivation is the reason or reasons one has for acting or behaving in a particular way. The fear of being executed is a good reason to act or behave in a particular way. I am not saying its right but it should be effective.
It is true that keeping the fighters in line in this way motivates them to follow orders. But this is only in cases where the fighters are stuck there, and there are large numbers in that position. Then there's the question of people continually joining. It would cripple the flow of fighters in the group because nobody's gonna want to put themselves in that situation in the first place.
Who in their right mind would ever be willing to fight for someone who just killed their buddy for little to no reason? Of course, most ISIS fighters don't have right minds anyway, but even for them, they have to realize that you don't motivate people to fight FOR you by killing their comrades, that's how you motivate them to unite AGAINST you.
Since the debate is regarding the effectiveness of a certain motivational tool rather than the ethics of the tool or the group utilizing it, I would say that no, it is not an effective technique. Anecdotal and statistical evidence points to positive reinforcement being more powerful than negative reinforcement or punishments in a number of environments.
No, while ISIS executing its own fighters is extremely effective in frightening its enemies and displaying its message, I do not feel that it is an effective motivational tactic. Potential members of the radical group may be frightened away by such actions. Also, even some radicals who share many beliefs with ISIS may find these tactics as wrong.
ISIS has finally made a disastrous turn, which should turn off those that are mesmerized by them. Many of the fighters gravitating toward them appear to be disenfranchised young men looking for something to belong to. This is a way for them to become a part of a brotherhood. For these same men to see that "family" turn on itself, it must feel like a bucket of cold water being poured on them. Hopefully this is the dose of reality they need and will wake them up to realize how dangerous ISIS truly is.