Everyone has the right to believe in religion they choose without being prosecuted or discriminated against, but in the privacy of their own home or church. Putting religious displays on public property can make people uncomfortable if it is not their own religion. Putting religious displays also violates the separation of church and state. Religion should have nothing to do with the state, government, or public property, and displaying certain religions shows more government/public support for a specific religion as opposed to being open for all religions.
Many of the original states were founded by specific faiths. The Declaration of Independence even references a general belief in God, saying that men were given rights, "endowed by their Creator." Because of the founding of the United States on religious principles, it makes sense that certain public properties might contain religious displays or monuments. The government worked to give Americans freedom of, not from, religion.