Total Posts:6|Showing Posts:1-6
Jump to topic:

Distractors in Multiple Choice

Logic_on_rails
Posts: 2,445
Add as Friend
Challenge to a Debate
Send a Message
2/17/2012 7:27:34 PM
Posted: 4 years ago
The reason one has tests is to try and understand whether a student has competency in the areas being tested. To this end, a test is successful when it successfully provides information about what a student knows and what additional instruction will be required to achieve the desired outcomes.

Traditional multiple choice tests provide little information about a student's level of understanding beyond whether he or she has selected the correct response. In multiple choice we have 2 types of answers to a question - the key (correct answer) and distractors (incorrect answers that typically represent common errors to those without the understanding to recognise the key as correct) .

The thing is that the current system can be improved - have distractors also identify the nature of a student's misunderstanding. This is accomplished by having a rationale for the distractor - an explanation of why that distractor is the wrong response. Obviously, the rationales in a taxonomy (effectively, a list of rationales) can be altered between subjects.

Let me give an example (from http://www.pearsonassessments.com...) of why have distractors that identify the point of error is useful:

"Table 2. A distractor rationale taxonomy for mathematics items.

Level 1

Makes errors that reflect combinations of conceptual misunderstanding and unfamiliarity with how to relate operational procedures to problem contexts. Student attempts to implement strategies that are unrelated to the task at hand. These errors may indicate that the student has an inordinate dependence on information that is explicitly stated in item stimuli, and is lacking the sophistication to correctly answer the question being asked.

Level 2

Makes errors that reflect some sophistication and computational ability, but that demonstrate an inadequate conceptual framework and flawed reasoning in support of conclusions or inferences.

Level 3

Makes errors that reflect definite levels of sophistication in analysis and conceptual knowledge, but that are flawed by inconsistent reasoning or computational weakness.

Level 4 - Correct response.

Janice spent $222.46 before tax to buy a climbing rope that sold for $3.50 per meter. What is the greatest number of meters of the rope she could buy at that rate?
A 7786.1 m [Level 1: incorrect operation with place-value error]
B 778.61 m [Level 2: incorrect operation, correctly applied]
C 635.6 m [Level 3: correct operation with place-value error]
D 63.56 m (correct) [Level 4: correct response]

Vs.

Janice spent $222.46 before tax to buy a climbing rope that sold for $3.50 per meter. What is the greatest number of meters of the rope she could buy at that rate?

A 0.6356
B 6.356 m
C 63.56 m (correct
D 635.6 m

All distractors are essentially level 3 in this example. Each is based on a computation error related to place value. The distractors in the second example each represent an error at the same level of understanding and do not provide any further information about the student's ability to perform tasks requiring this mathematics skill."

Essentially, changing the use of the distractors provides additional feedback from tests which thereby makes tests more useful.

For anybody interested check out this link http://www.pearsonassessments.com...
"Tis not in mortals to command success
But we"ll do more, Sempronius, we"ll deserve it
F-16_Fighting_Falcon
Posts: 18,324
Add as Friend
Challenge to a Debate
Send a Message
2/17/2012 9:44:46 PM
Posted: 4 years ago
I found this post very interesting as I am about to take a standardized test for Graduate school in the next couple of weeks. For the most part, I actually agree with your analysis and would like to see your proposal implemented in tests. However, let's discuss the potential problems.

Through your analysis as well as the linked source, we can see that there are four levels of understanding. It is apparent that reaching a Level 3 understanding for any question requires a lot more effort and preparation by the student than a level one response. This effort needs to rewarded and the students who reach level 3 should be given more points than those that reached level 1. The reason for this is that as you say, tests are meant to evaluate the student's understanding of the material. A student who checked the level 3 answer has more understanding than a student that checked the level 1 answer. So, the only way to measure this would be to grade multiple choice answers not as right or wrong but give a specific number of points for each answer. For instance, selecting the right answer would get you 5 points, a level 3 answer gets you 3 points etc.

The current system while flawed can measure a student's understanding better than the proposed system as long as multiple choice questions are graded as right/wrong due to the fact that all the distractors are at the same level.
Logic_on_rails
Posts: 2,445
Add as Friend
Challenge to a Debate
Send a Message
2/18/2012 1:43:45 AM
Posted: 4 years ago
At 2/17/2012 9:44:46 PM, F-16_Fighting_Falcon wrote:

Through your analysis as well as the linked source, we can see that there are four levels of understanding. It is apparent that reaching a Level 3 understanding for any question requires a lot more effort and preparation by the student than a level one response. This effort needs to rewarded and the students who reach level 3 should be given more points than those that reached level 1. The reason for this is that as you say, tests are meant to evaluate the student's understanding of the material. A student who checked the level 3 answer has more understanding than a student that checked the level 1 answer. So, the only way to measure this would be to grade multiple choice answers not as right or wrong but give a specific number of points for each answer. For instance, selecting the right answer would get you 5 points, a level 3 answer gets you 3 points etc.

An interesting point. Let me give you an analogy though. A bridge is constructed at what could be called a 'level 1' understanding. It falls into the water below. A better bridge is constructed at a 'level 2' understanding. It doesn't matter that it's better constructed - it still falls into the water. The bridge will keep falling into the water until it is constructed properly.

Also, these levels incorporate the previous style of distractors (all level 3 in the example provided) , just less of them. So, in the current system all answers level 3 or lower are deemed worthless. Why would the inherent value in a new system change?

I think however (aside from the bridge analogy) that the strongest point is merely that even if the marking is less fair (given the current marking system ... that's another variable to consider) the aim of the test is better achieved - namely, better, more detailed feedback to the teachers. The test betters achieves what it sets out to do. Is this a bad thing?

The current system while flawed can measure a student's understanding better than the proposed system as long as multiple choice questions are graded as right/wrong due to the fact that all the distractors are at the same level.

I don't quite get this last part. If both systems grade as 'right/wrong' then both will produce at least X feedback. The thing is that the distractors should produce X+Y feedback.

Your problems with the idea are definitely noteworthy. I do look forward to any potential further counters.
"Tis not in mortals to command success
But we"ll do more, Sempronius, we"ll deserve it
F-16_Fighting_Falcon
Posts: 18,324
Add as Friend
Challenge to a Debate
Send a Message
2/18/2012 10:22:47 AM
Posted: 4 years ago
What I meant by the last part is this: Currently we have multiple choice questions with distractors all at the same level. Let's say they are all level 3 for the sake of this argument. The correct answer is at level 4. The question tests whether a student has reached level 4 understanding or not. If they have reached level 4 understanding, they get the points. If they didn't, they don't. I agree with you that this can be flawed. My point is that while I like your system, it doesn't address the flaws unless you change the grading style for each question to reflect the amount of work needed to get each answer. Let me explain:

Here's how I would look at the bridge analogy. The main difference is that questions in tests measure learning whereas construction of the bridge measures actual performance. Let's say you are the head engineer looking to hire 10 people to build a bridge for you. 20 applicants apply and you set a test for them. The questions are all multiple choice. You make the test exactly as you proposed with level 1, level 2, and level 3 questions. Now, let's say you get 8 people who you clearly want to hire and 4 people who are borderline and you need to pick 2. They all have the same score. However, you want to hire the ones with the most understanding on how to construct the bridge. Who would you hire?

It is here that the issue of how to grade the questions comes in. If there were two people who demonstrated a higher level of understanding by checking level 3 answers, those are people who you want as opposed to the ones that checked a level 1 answer. There needs to be a distinction made between those that checked level one answers and those that checked level 3 answers.

My point is this: Your system is good, but to get any real benefit from it, you need to find a way to reward and distinguish those who had higher level answers from those with lower level answers by giving a different number of points for each answer. Without such a distinction, what really is the benefit of your system over the current one? It doesn't help gauge understanding better if the scores remain the same. It doesn't help evaluate students better if the scores are all the same.

This is an interesting discussion, I must say. I look forward to your response.
Logic_on_rails
Posts: 2,445
Add as Friend
Challenge to a Debate
Send a Message
2/18/2012 4:41:18 PM
Posted: 4 years ago
I'll try and not quote to reply (I normally do) .

Let me reiterate what I feel is a key point - tests are designed to give teachers feedback on student understanding. The more (or better) feedback given the better the test is. That's where much of my case is derived from.

To address your counter to the bridge analogy counter, obviously the students who demonstrated level 3 answers are more knowledgeable. However, this doesn't make them sufficiently knowledgeable. Maybe their lack of knowledge will force a closure to the bridge building project. Perhaps the test designer will see that his written test is lacking (test design in general is not perfect) and make seek to evaluate understanding in other ways (perhaps an interview). The point is that the current system already presumes that understanding below perfect is not worth anything (level 3 or lower gets no marks) . My question is 'why should this change under the current system?' The only difference is the style of the questions. One need not change the way one values certain degrees of understanding.

I understand we are likely looking at this from different perspectives / focusing on different issues of concern. I'll use this quote of yours to try and demonstrate my point:

At 2/18/2012 10:22:47 AM, F-16_Fighting_Falcon wrote:

My point is this: Your system is good, but to get any real benefit from it, you need to find a way to reward and distinguish those who had higher level answers from those with lower level answers by giving a different number of points for each answer. Without such a distinction, what really is the benefit of your system over the current one? It doesn't help gauge understanding better if the scores remain the same. It doesn't help evaluate students better if the scores are all the same.

Your point is that benefit in understanding student knowledge can only be had if the scores are differentiated. I must respectfully disagree. Even if the exact same marking method is used teachers can still look at questions and say 'ah, he demonstrated this type (or lack of) of understanding. That means I need to cover this point' . Whereas if the point of confusion is unknown (as with similar level distractors) then the teacher must address all potential issues in order to be assured of addressing student difficulties. If the answers in a test show that students don't suffer from all potential difficulties then the teacher has saved time.

I believe our point of disagreement to be that I am focusing on the feedback given to teachers while you are focusing on the feedback to students not differing (at least not obviously) . Both are important obviously.

I look forward to a potential response.
"Tis not in mortals to command success
But we"ll do more, Sempronius, we"ll deserve it
RoyLatham
Posts: 4,488
Add as Friend
Challenge to a Debate
Send a Message
3/13/2012 9:49:20 AM
Posted: 4 years ago
If the alternate choices reflect common errors, then the test will provide useful feedback on what mistake the student is making. If 90% of the students who err do so by misplacing the decimal point, then it seems to me appropriate to have the distractors representing that ype of error. If other types of misunderstanding are more common, then that ought to be in the mix.

A programmed learning system is probably a better mechanism than a straight test for diagnosing the nature of misunderstandings. Programmed learning system can accept answers other than multiple choice, and also when the answer is wrong the system can go into more detail on analysis of the problem.

Multiple choice questions are designed for machine grading, which is a limitation.