The perpetrators, society, nature, the victims. All deserve at least part of the blame on a case to case basis. This is not just regarding rape but also includes victims of theft, mugging, extortion, bullying, natural disasters, war, etc. As for victims, people should learn to take responsibility for themselves and whatever the consequences are of their actions (read the fable by Aesop of ancient Greece: "The Farmer and the Viper").
The only people who are ever to blame in the case of a rape case is and will always be the rapist however, with standard sexism and the need to protect men's reputations at all costs, women's pleas are drowned out in a chorus of insults from people calling her a slut, whore, or jealous attention seeker. People don't seem to grasp the emotional toll that admitting that you've been raped has on a victim. Furthermore, when the justice system tramples their trust in the system, it only adds to their sense of hopelessness and distrust when no one believes them. (Watch the Hunting Ground on netflix. It really highlights the victim blaming aspect of sexual assault.)