Yes, wife swapping could have a negative impact on the self-esteem of those who participate in it. If one of the spouses uses the great qualities of the spouse they swapped with against their spouse it could hurt their self-esteem. They could use it to shame their spouse and make them feel inadequate.
It could have that effect, but it also easily couldn't. Poly-amorous relationships are somethings that all parties involved need to be ready for. Jealousy and cheating are often easy results, and they can be manifest in people who thought they were willing to try the swinger lifestyle at first. Like with anything, couples need to talk and figure out what they truly want.
Wife swapping could most definitely have a negative impact on a person's self-esteem. The intimacy between a wife and a husband is based on the confidence that each will have the other's back, guard their secrets, and overlook their faults. Wife swapping would attempt to bring in the intimacies of a relationship without the prep work that needed to be done beforehand. A new partner would not know how sensitive a person was to certain words that indicated childhood wounds, for example, or that criticizing a person's habits or idiosyncrasies might cause them great embarrassment.
I believe that wife swapping could have a negative impact of those who participate in it. Upon return home after the swap it is likely that the spouses will compare the actions of their spouse and the swapped spouse. A man might be harsher on his working wife who does not have time to cook dinner each night and keep a perfect house after having experienced a stay at home mom who has time to clean and cook all day.
I believe that a stable happy marriage has a massive positive effect on self-esteem, and that anything that can help a marriage stay in place is a very positive thing. Many people get bored with the routine and predictability of their marriage, and wife swapping is one thing that might help keep things exciting, and help the couple stay together, therefore I think it can improve self-esteem rather then lower it.