If we censor the symbols that capture our attention, how will we be able to learn their messages? Take the Nazi symbol. We seem them as the ultimate evil (though I would argue that Stalin and Mao Zedong were more evil, just look at their kill counts, especially the latter. Mao makes Hitler look kind by comparison), but why? They attempted genocide. The Nazi symbol makes us remember their actions, the atrocities they committed. It serves as a warning to not repeat the mistakes of the past.
But what about more modern examples? Well, what can be considered offensive nowadays? Simply disagreeing with someone? Actually, yes. Modern Social Justice circles dismiss and shame any opposition as racist/sexist, sometimes even deserving of death. They're offended by contrary opinions. Should anything like that be censored?