I grew up in America and I have been drinking cow's milk all of my life. I've never asked myself if I thought that it was ethical. It comes naturally to me and it also tastes pretty good. Not to mention the health benefits. I do not believe there's any reason to think that it would be ethically wrong to drink an animal's milk.
It is my opinion that drinking an animal's milk is ethical because it is a very natural thing to do. Human beings are considered animals, and the natural way for babies to feed is to nurse from the mother's breast. Therefore I believe that drinking an animal's milk must be an ethical practice.
Drinking animal milk can be ethical if the animals are treated ethically. Farms that never allow animals to see grass and are stuck in cages should be boycotted. Other, ethical growers and producers of milk that treat animals with due respect and morals are fine to buy milk from. Meat producers is another argument altogether.
There is nothing wrong with taking advantage of our surroundings. We eat a chickens eggs and eat many other animals meat, this is no different than drinking milk from an animal. Milk is a very good source of energy and it will probably never be thought as unethical. We use animals for food, it won't change anytime soon.
I see nothing at all unethical about drinking the milk of an animal. Humans have been doing it for centuries, primarily with cow's milk. Other animals include goats, yaks and even sheep. It provides nutritional benefits. And there always seems to be enough for future generations of livestock to continue.