Maybe the sciences are pretty complex as they explore our world to a huge depth but it is not proven to be infinitely complex. Whereas mathematical equations can be expanded infinite times. Mathematical equations can also stack upon another to create even more complex equations. Mathematical ideas can be interpreted in infinite ways as well. And also applied to many different subjects like the sciences.
For some people mathematics is very easy. While for mathematicians other subjects are immensely hard. In the same way an understanding of literature is something that can escape many people. As such every subjects difficulty is subjective to the person doing it. As such Mathematics may certainly not be the most difficult subject in the educational system.