Passing judgment on people based on physical appearance is unfair to both you and the other person. Not only can you miss out on getting to know interesting and great people, but there may be circumstances contributing to people's appearances that are beyond their control. Furthermore, if the concept of immoral stems from a religious perspective, those who follow Judeo-Christian principles are taught that they should love all things.
Fair or not, physical appearance plays a big part in how humans understand and interact with each other. Judgments drawn from appearance should not trump those drawn from conversation or behavior, but they are still useful in first impressions and general assessment. Such judgments shouldn't be quantifiable (good or bad), but, rather, qualifiable (this way or that way, neither being better or worse) so that they give an idea of one part of a person's character.
It is a natural reflex to start making judgments and assumptions about someone just based on how they physically look. What would be immoral is to totally judge them based on physical appearance. It's just that until you have a chance to get to know someone personally, you really have nothing else to go on. It's okay initially.
Judging someone based solely on personal appearance is not really a question of morals, it is a question of logic and rationality. Plenty of people that make strong moral choices judge by appearance because it is part of human nature. A logical and rational person does not use appearance as the sole benchmark of judgment.