The death star can only fire in one direction and takes an absurd amount of time to maneuver into firing position. Let's not forget that people on the enterprise have the ability to teleport. They can just beam onto the ship, plant a few high grade explosives in the right spot, then just warp out!
Enterprise v Death Star is no contest. Phasers, photon torpedoes, and as medicfrog says, they have transporter technology, a huge competitive advantage over the Death Star. They wouldn't even have to beam themselves aboard the Star, they would just have to beam over a few hydrogen bombs on timers. The Enterprise is a highly maneuverable ship. Where is would have a challenge is with the clouds of Tie fighters and other smaller fighting vessels that the Empire seems to have in unending supply. Lucky they have shields. Bottom line, failing all else, the Enterprise has Spock, who can think and pinch his way out of any jam.
Even if the death star has a bunch of smaller ships in it's cargo bays and whatnot, the enterprise phasers have never been shown to have a muzzle on the enterprise, so the could shoot 360 degrees, towards any tie fighter of tie advanced. And really, teleportation just makes it more than op.
The Death star is a massive space station that can destroy a planet and has only one weak spot which can only be punctured with a proton torpedo, and is located at the end of a long trench which only a small fighter could enter.
The enterprise is A) Too large to skim the trench, and B) doesn't have Proton torpedoes.