Sexual harassment is the most harmful of types of harassment because it not only incorporates the language used but can also include physical contact, emotional trauma and also ridicule from co-workers either in favor of the harasser or angry at the victim for not correcting the situation. Harassment of any sort should not be tolerated, but sexual harassment can affect the entire working environment.
Yes, sexual harassment is the most harmful type of harassment, because a person is so confused when it happens to them. The person might be ashamed, or they might not even fully appreciate what is happening to them. Sexual harassment puts a person in a bad situation, because they do not want to lose their job.
If you are comparing cat-calling to stalking, then no.
If you are comparing constantly propositioning someone while stalking them around the neighbourhood and harassing them with unwanted intimate images to the point where the person does not feel safe in their home or anywhere, to someone calling you names you don't like at work, then yes.
It is the severity and context of the harassment that determines the impact, not the type of harassment itself. It also shifts from individual to individual.
Sexual harassment can be very damaging, or it can be merely inconvenient and irritating. The are other types of harassment which can be just as damaging, even more so, if the harassment is more intense, and more frequent. I'm thinking of harassment based on religion, or perhaps, racial harassment. It all depends.
In my opinion any type of harassment is pretty bad. Harassment alone induces anxiety which can be quite tormenting on its own. Sexual harassment really depends on the individuals involved, some people call it out while others let it slide by. It's certainly up there, but I wouldn't say it is the most harmful.