Saying stuttering is annoying is a bit rude actually. Stuttering is a speech problem that in some cases requires medical attention. It's not annoying, it's a speech problem that should be respected, not called annoying.
What Sandy says can also be applied to not saying anything. I am autistic, and I sometimes have a hard time saying what's on my mind. However, I can understand how it would be frustrating to someone who needs my reply.