Feeding calves nothing but milk isn't a way to raise baby cows just so they can have a certain cut of veal. At a certain point, calves are naturally weaned from their mother's milk and they start to eat grass like any other cattle. Veal may as well be the foie gras for beef as cows are force-fed milk instead of more solid foods.
I would say that a milk only diet, or any kind of diet for that matter, for calves or any other creature bred to be slaughtered, is not inhumane. This is assuming, of course, that it does not actually physically harm them, any more than vegetarian humans are physically harmed.