If artifacts are taken away from cultures wrongly, then they should be returned to that culture. For example, if Native American grave sites have been ravished and items taken, there should be a conscious effort to return those items to that culture. Some may say that if they purchased the items then they are theirs. However, it is in the interest of an art buyer to make sure that what they are buying from a reputable dealer. Historical objects should never be taken away from a culture.
Many of the items would be vandalized. For example, in Syria, while the country was in military crisis, the ISIS destroyed many important historical items, even a temple. While destroying this hundred-year old landmark, they were celebrating. If there would have been a way to bring the temple to a museum in London or New York, the temple would have been preserved and would not be in the disrepair it is in now.
Words of art were usually bought with consent of their rightful original owners. At any rate, the people who discovered them and paid to have them excavated and restored have more of a right to have them than people who just happened to be born in a country with a long history.