Felix Qui Potuit Rerum Cognoscere Causas - Anjel Lertxundi

Science in Action

A Question of Choice -

Multiple   Choice


Are you a student facing a multiple choice exam soon ? A teacher tempted to give your students multiple choice tests ? Got to pass a multiple choice test such as an IQ test or something similar? Do you know what you are getting yourself in to ?
Science in Action : Probability and statistics in Multiple Choice, or how to beat the odds and pass a multiple choice test effortlessly (or : a fair way of preventing your students from doing so).

  1. Why people like Multiple choice
    1. student's point of view
    2. teacher's point of view
  2. Guessing and Gambling on multiple choice tests
    1. Does it pay to gamble ?
    2. teacher's counter measures
  3. Grading and Scoring on Multiple Chopice tests
    1. Average Gambling Score
    2. Probable Gambling Score

The Student's point of view

Students often prefer multiple choice tests because they think it's easier then open questions. They count on the fact that they can recognize the correct answer when they see it, while formulating a correct and complete answer from scratch (or from memory) is conceived more difficult.

However, a skilled teacher can formulate the options of a multiple choice question so, that you really need to know your stuff in order to pick the only 100% correct and complete answer.

The Teacher's point of view

Some teachers prefer to give their students multiple choice exams because they're easy to mark : an answer is either correct, or not - they don't have to read the answer and evaluate it, they only have to compare the answers to their list of 'correct answers'.

However, this only goes if the questions and answers are well formulated. If the student that takes the test interpretes the question differently from what the teacher intended, he may give an answer that is correct in his interpretation of the question. The teacher may consider this answer 'wrong', because he does not realize a different interpretation of the question is possible, or because he assumes that the student lacks the knwoledge that would allow him to interprete the question in a different context.

In a highschool Physics exam, the teacher may ask questions and propose answers in an implicitely assumed Newtonian context, unaware that a student may have read and understood a few things about Einstein's relativity theory or about quantum mechanics. Questions about speed, position, acceleration, sense and direction of movements etc. can have several correct answers, depending on whether you take Newton's point of view, Einstein's curved space, or the weird world of quantum particles as a starting point.

Guessing and Gambling

Multiple choice allows the students to gamble. Even if they don't know the answer to a question, they know the odds (1 out of 3, 1 out of 4, ...) of picking the correct answer by chance. They may also reason that, over the whole test, there will be a more or less equal distribution of A, B, C, D, ... as correct answer : every answer has equal chance of being correct, so that if you're sure about some answers and count how many times the correct answer was A, how many times B, how many times C, etc., you can easily fill in the rest. Like : if 'answer C' clearly has not yet been choosen as often as A and B, you may prefer answer C for the questions you don't know, except for the questions where answer C is clearly nonsense.

In some tests, students may notice that the answers for each question are not really in random order, but that the person who designed the test, unknowingly, applied a system to provide answers to the questions. Maybe A and B are usually the correct answer, as if the teacher wrote 1 correct and 1 obvious wrong answer to each question he came up with, and added two (wrong) answers (C and D) afterwards. In this case, gambling between A and B for questions you can't answer gives better odds (1 out of 2) than random gambling (1 out of 4).

An other system that may have been applied to the answers is somthing like

So here, 'C' would be your preferred bet for answers you don't know the answer to.

Of course, this is all a bit 'twilight zone' : perhaps, as a student taking the test, you think you see a system, but there is none. Then all your guesses will be based on wrong assumptions. Or your teacher knows you'll be looking for a system so he pretends to be using one but doesn't, or orders the answers so that your understanding of the system will lead to incorrect answers. It becomes a ''who's fooling who?'' situation.

Countermeasures to gambling and guessing

To avoid all this guessing and gambling, teachers will

It's this ''subtract points for wrong answers'' that is often misunderstood. Teachers often pick arbitrary, but rather high numbers as a deterrent for gambling, like ''each correct answer is worth 5 points, and I subtract 3 points for each incorrect answer, so if you're not sure about your answer, you better leave it blank in stead of guessing'', or even worse : ''a correct answer = 1 point, no answer = 0 points, a wrong answer is -1''

Sounds fair : if you don't guess, you'll just get all the points for the correct answers you've given. But what if you don't gamble, but still answer wrong : maybe you've miscalculated something, or remembered wrong, or so. People do make mistakes. And your wrong answer will make you loose points that you rightfully earned on other questions.

This does not sound so fair any more. Say that, out of 20 questions, you answer 13 correctly. With 5 points to each question, you'd pass the test with 65/100. Not too bad, that. However, you loose 7x3 = 21 points (7 wrong answers, 3 points lost at every wrong answer) on the wrong answers. Suddenly your score drops to 65-21 = 44/100, and you fail spectacularly.

The mathematical approach

On a multiple choice test where every question has 4 possible answers, completely random answers (i.e. a student ho doen't know a thing, and answers randomly to each question), will - statistically- result in a score of 25%. Statistically, that is : if, say 100 000 students gamble on this test, their average result will be a somewhat near 25/100.

The thing with averages is that they even out the extremes : among those 100 000 students, some will score 0 or 1/20, while some may score close or over 50/100 and pass the test. There's even a small change that a student guesses all answerts correctly.

A small example : consider a test with only 2 questions, and 2 possible answers (A or B) to each questions. When filled out randomly, there are 4 possible results :

When 100 students take this tests, and they all gamble (give random answers), 25 will have answered as in result 1, 25 others will have given result 2, etc. Now lets consider the outcome. If the correct solutiuon to this test was result 4, the 25 students who (by chance) answered 'B' to each question will score 100% (2 out of 2) on the test. 50 other students (25 with result 2, 25 with result 3) will score 50% : they've answered 'B' to 1 question. So if the minimum score to pass for this test was 50%, 75 out of 100 gambling students will pass. The probability of passing (by giving random answers) is 75/100.

It stands to reason that a teacher will not want to see 3 quarters of the class pass a test without studying, so he'll want to discourage gambling by subtracting points for incorrect answers. The point is : how much is fair ?

Going for average gambling score = 0

The easiest approach to this problem is : make sure that a student who gambles on all questions, will score - on average - 0 points.

Consider a question with 4 possible answers. A correct answer is worth 3 points. A gambling student has 1/4 chance to guess the correct answer, and 3/4 chance to guess wrong. Thus, if 1 point is subtracted for a wrong answer, this evens out : on a larger number of questions and/or a larger number of studens, the average score of the gambling students will be near 0 : for each chance to win 3 points there are 3 chances to loose 1 point.

The rule to apply here that the points to subtract for wrong answers is related to

  1. the number of given options (possible answers) per question
  2. the amount of points to be won with a correct answer

The formula : with
Pi = points to subtract for a wrong answer
Pc = points to be earned with a correct answer
A = number of multiple choice answers to a question

Pi = Pc /(A-1)

It works best when all questions have an equal number of options, and the points to be earned with each question is the same for each question. If this is not the case, you can use the total of points to be won on the whole of the test, and the total of incorrect answers in the test. The average will work out roughly the same.

This is a first, rather straightforward approach to dealing with multiple choice : on average, a gambling student will score 0 points. That's fair to me.

It's also fair in a realistic scenario : a student will study, and take the test with at least some knowledge. Let's assume he answers 6 out of 10 questions, convinced that they are correct. If he leaves it at that, he may score 6/10, or 18 out of 30. If he guesses at each of the remaining questions, he's likely to guess 1 correct, and 3 wrong (if there's 4 options to a question). So he'll win 3 points, and loose them again with his wrong answers, and his score remains the same : guessing doesn't work. If he only answers 1 or 2 questions, there's 3 times more change to loose 1 or 2 points (so he'll fail) than there is to win 3 points.

The odds of passing a multiple choice test, without studying

As illustrated earlier, averages even out extremes. A statistician had to cross a river. Knowing the average depth was 2 ft (60 cm), he thought is was save to wade through it. He drowned. Even if the average score of the gambling students is near 0, there may be some who score far better than that. Maybe even enough to pass. That would depend on the number of questions, and the number of answers, as illustrated in the 2 questions, 2 answers case, where a student would have a 75% probability of passing the test by random guessing.

The question here is : how many different ways are there to complete the test, and who many of those result in a score that is high enough to pass.

In the simple example earlier, we saw that 2 questions with 2 answers each can be answered in 4 different ways. This is the 'population' : the total of situations to consider.

With
s = 'space', possible situations
Q = number of questions
A = number of possible answers to each question

P = A ^ Q (i.e. A to the power of Q)

So, as we've seen, 2 questions with 2 possible answers each give 4 possible sollutions, 3 questions with 2 answers each give 2 ^ 3 = 8 possible results, 3 questions with 3 answers each give 27 situations, etc. The probability to pass by guessing is the answer to the following questions :

  1. How many questions do you need to answer correctly to pass ?
  2. In how many of all possible results (the population) does this occur ?

The first part, ''How many questions do you need to answer correctly to pass ?'', is arbitrary. It is a given : the teacher (or the school, or the ministry of education) decides that you need to score 50%, or 60% or whatever, to pass. And the teacher may also decide how many points to win per question, and maybe also how many points you'll loose for an incorrect answer.

With these givens, you can deduct that
with
Q = number of questions
n = points needed to pass te test
t = points to be earned with a correct answer
f = points to subtract for a wrong answer
c = number of correct answers given by student
w = number of incorrect answers given by student

a student's score = the points earned with correct answers - points lost for wrong answers
score = c*t - w*f
where w = number of questions - correct answers = Q - c
so that, in order to pass, the folowing statement must be true

ct - (Q-c)f = n

Now use simple algebra to calculate c, the (minimum) number of questions a student has to answer correctly to pass the test, taking into account the points deducted for incorrect answers.

ct - (Q-c)f = n

=> ct - Qf + fc = n

=> ct + cf = n + Qf

=> c.(t+f) = n + Qf



=> c = (n + Qf) / (t + f)

As this will probably be a decimal (real, not integer) number, so it will need to be rounded up, because it represents the minimal number of questions to be answered correctly in order to pass.

c = INT ( (n + Qf) / (t + f) ) +1

How often does this happen ?

We have found how many possible ways there are to solve a multiple choice test, and what the minimal score is to pass. Next question is : how often does this happen : how many of the randomly filled out tests will show a passing score.

To examen this question, lets have another look at a simple situation, like a test consisting of 2 questions, each with 3 possible answers. There are 3 ^ 2 = 9 ways this test can be filled out.

For the sake of argument, let's assume that the correct answers were A for question 1, and B for question 2. You'll find that in situation 2, that's the answers our gambler will give. This situation occurs once out of 9; the probability of scoring 100% correctly on this test (by pure guesswork), is 1/9 or 11.1 %.

Any other situation were you answer B to question 2 OR A to question 1, you score half of the points (and pass - we're not deducting points for wrong answers here). This happens in 4 situations (situations 1, 3, 5 and 8) out of 9.

Everything added up, there are 5 out of 9 situations where gambling results in a passing score of at least 1 out of 2 questions right. That's a probability of 5/9 or 55.5%. This means something like : if 1000 students take this test without knowing any answer, 555 will still pass. Those are betting odds - for the sudents. Note that, because we take a general approach to all possible situations, the odds remain the same for every combination of 'correct answers'.

Of course, no teacher will give a test with only 2 questions, 3 answers each. And as the number of questions increases, it gets more complex to write down all possibele situations, and count the 'winning' situations to calculate your odds (see graphs at the end of this paper). You need a formula. The teacher will also introduce a rule such as ''3 points for a correct answer, -1 for a wrong answer'', which makes counting the winning situations also rather complicated.

We've already deduced a formula to tell us how many answers need to be correct, and a formula that tells us how many situations there are to consider. The only thing we still need is a formula to show how many times a 'winning situation' will show up in the total collection of possible situations, because those are our odds of passing the test without knowing 1 answer.

We've established that, as we're considering every possible situation, it does not really matter which really are the correct answers. Say you've established that you need to have 3 answers right to pass a test of 6 questions. There are a number of ways to get at least 3 out of 6 right : You can get 3 out of 6, 4 out of 6, 5 out of 6, and 6 out of 6.

There's only 1 way to get 6 out of 6 : you have to get al questions right : 1-2-3-4-5-6. To get 5 out of 6, there are already a number of ways : 1-2-3-4-5, 1-2-3-4-6, 1-2-4-5-6, etc. This is the subject of the mathematical theory of combination and permutation. The number of possible combinations and permutations can be calculated by factorials. The formula for the number of combinations of k elements out of n is

n! / ( k! * (n-k)! )

Since we are interesting in 'at least' k out of n, we have to repeat this for k+1, k+2, k+3 etc.until k=n, and add up the results. The theory of binomial probability teaches us that we also have to consider the chances to succes or failure on each question. E.g. when there are 4 answers given to each question, and only one is correct, there is 1/4 chance to success against 3/4 chances for failure. That's a lot of calculations, so we'll get a small program to do that for us. Most staticicians do.

So, what are my chances then ?

graph : drop of odds with increasing number of questions
Here's a graphical representation of how chances drop while the number of questions increases, for the 3 scenarions (without loosing points for wrong answers, loosing points in a fair way, etc). Especially at when the number of questions is limited, there is an additional disadvantage ad even and uneven numbers : e.g. to score 'at least half' on 3 questions, you need to score 2 out of 3, while with 4 questions, you need to score only 2 out of 4. It's obvious that the latter is easier. That accounts for the curves.

Your chance to guess r questions right on a multiple choice test with n questions, each with a answers given is

C(n,k) * p ^(k) * q ^(n-k)

where
C(n,k) = n! / ( k! * (n-k)! )
probability for success = p = 1 / a
probability of failure = q = (a-1)/a

Throw it all in a bowl, add a grain of salt, and you'll find that the probability to score 5 out of 10 on a multiple choice test with 4 options to each question is 7.8 %. Adding the disadvantage of points lost for wrong answers - say the fair deal : 3 points for a correct answer, -1 for a wrong answer - , reduces your chances to 0.35 %. Sound's OK to me; I mean, no student in his right mind is going to gamble 1:300 to pass a test. There seems to be no reason for even harder punishment, such as 1 point for a correct answer, -1 for a wrong answer (which reduces the probability of passing to 0.041 % or 1:2500, by the way), as we have demonstrated earlier that the 'average for gambling = 0 points', while it may not keep the students from guessing, it will keep them from gaining anything from it.

Furthermore, because of the exponential effect (total number of possible sollutions increases exponentially as number of questions increases), the probability of passing by gambling drops dramatically (less then 0.0001 %) when the number of questions reaches 15 or more (see graphs). So adding questions to the tests is, for the teacher, a better strategy than subtracting points for wrong answers.

Epilog

Now, If I spend some time on studying, in stead of developing this theory on multiple choice tests and calculating the odds of passing without studying, I would pass that test for sure. I guess.

Graphs

The probability of passing a multiple choice test depends primarily of the number of questions, because, as the number of questions increases, the possible solutions increases exponentially, and therefore your chances drop accordingly

While 3 or 4 questions with each 3 opr 4 possible answers still give rather good odds,
exponential growth - graph 1
the number of possible sollutions explode between 15 and 16 questions : student is left without a chance :
exponential growth - graph 2


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. For attribution, include a link to this page.