Assuming the question here is accurate, I find it pretty reprehensible that there even are "special privileges" to begin with. Education is a serious matter, not something to be left up to who can or can't pay a particular fee. More people educated in the highest places possible can only be a positive.
No, the college fee and special privileges of Oxford and Cambridge should not be abandoned, because it is historical. If students can look forward to being a part of something that is truly historic, they will work harder for good scholarship. Oxford has a long and proud tradition and there is no reason to change the system that has worked for many years.