Yes, manned flight is bad for the environment and in combinations with many other things, ruining our planet, but it only plays a small part in that. We do not have many manned flights, but it is true that some rockets use toxic propellants or carbon neutral propellants and other chemicals. These cause holes in the ozone.
Although manned flight is doesn't do great things for our carbon footprint, it is one of the smallest threats to our planet. The over usage of natural resources that cannot be replaced quickly, massive de-forestation that disrupts complex eco-systems and has even driven some of them to extinction and even the overuse of the automobile are all ruining our planet at a much faster rate than manned flight.