How to handle objections to the wording of test questions or answers?

Question

I try to include a wide range of difficulty in the questions on quizzes and tests, from very obvious to quite subtle. The more subtle questions might hinge on a distinction of singular vs plural or other 'wording' aspects, so that those who have the strongest grasp of the material can differentiate themselves. I usually get a good spread of numerical grades, so experience says that I am doing this correctly.

But students who missed an item based on subtleties often speak up when I review the quiz or test immediately after all the results are in. It would seem impolitic for me to say that I made that item difficult deliberately, but it is true. Instead I usually point back to the textbook or my lecture to say that I actually explained and even stressed the point. Sometimes I agree that wording was poor in a question and so I give everyone credit (essentially acting as if the question did not exist). This can vary from year to year.

In a workplace subtleties will be vital, so it is not a manufactured situation to include them in my teaching, and I only use subtle distinctions in tests where they exist in the material, and are worth emphasizing.

How do you handle situations where students push back on the wording of test questions or answers? How do you not simply say, "Because that way I have a more useful test"?

Answer 1

I am not sure what you mean by a "useful test" in this context. If your goal is to create a bell curve with English speakers at the top, you could administer the verbal section of the SATs.

You say that your students are complaining because what you actually wanted from them was unclear without an eagle-eye reading (and from your own accounts, it sounds like this is the case). If this is the case, then their complaints are legitimate. It is perfectly reasonable for them to be upset that you are trying to trip them up in a manner orthogonal to what they are actually studying.

I'd like to differentiate two different forms of testing. The first is meant to enhance pedagogy by plumbing the depths of a topic. A great pedagogic test creates an occasion for students to consolidate what they know, but it also provides a means of discovery. Almost everyone that I have ever spoken to about this can remember testing events when, after reading a question, they suddenly understood something that they had been missing about the topic itself. This sort of test simply allows students to demonstrate whether or not they understand the material, which forms a sort of philosophical boundary. If all of the students understand the material, all of the students will achieve close to top marks.

There is another form of test which is designed to rank students. The boundaries here are different, and there is some psychological brinksmanship involved. Even if all of the students understand the material, they will not all achieve top marks. There are a number of ways to accomplish this. One is to simply ask a lot of tiny, very detailed questions. Another is to use questions about the subject that don't even have to be particularly hard; you simply have to set enough traps. The best and most capable students fall for the fewest traps (or so the thinking goes). Of course, using this second method causes resentment.

But very fundamentally, the goals and means of these two types of tests that are fundamentally different from each other. By trying to combine them, you ultimately wind up with a test that is good at neither. And if students expect pedagogical tests that are designed simply to test knowledge, but instead they receive ranking tests, they will feel that their trust has been breached. When students do not trust their teacher, learning is dampened.

We should not resort to verbal trickery to differentiate between students. Make the questions harder by forcing them to integrate the material more deeply, or use it at a more advanced level. They can reasonably become confused about how to accomplish the difficult task you have set before them. They should never become confused about what that task is in the first place.

Answer 2

If you are to do summative assessment, then it is important not just that their are a wide spread of grades, but also that the correct people get the correct grades. Ensure that the questions differentiate correctly. That is high performing students get hard question correct, but everyone else (usually) gets them wrong. All but the weakest students get the easy questions correct.

If you can, do formative assessment. This means that the results, also have to tell you something about:

• Your teaching: what areas you need to improve / cover again.
• What pupils know: what areas they need to revise.

You should not be trying to catch students out.

Have a look at https://helloworld.raspberrypi.org/ issue 3. It has several article on assessment. See also How do you assess students' understanding of abstraction? it refers to project quantum; watch the video. Project quantum, not only assesses the students, it also assesses the questions. It wants to find questions that differentiate correctly.

After all of this. Show that the subtleties are useful. Show that these questions are correctly predicting current performance. Show that there is a strong correlation: Only high end students are getting it correct.

Answer 3

Since you are already willing to, in effect, make a question moot on an exam when there are objections you are partway to a solution. As ctrl-alt-delor notes here you need to assure the validity of the exam itself. It is possible that your interpretation of "subtlety" is actually misleading to students, especially the ones you think are "best" for reasons other than the exam.

But a better solution is to run a statistical analysis of the exam itself after the students take it. There are measures of consistency that will tell you whether the individual question marks are consistent with the overall marks. See Reliability measures here for example.

However, there is a simpler way that you can proceed, provided that you explain it to the students in advance. There is no absolute law or reason that every question on an exam has to count toward the overall grade. You can have a few ultra subtle questions on an exam that are not intended to "count" but which give you additional information you can use for, say, advising and shepherding. You don't need to tell the students which questions are the non-counters, but should let them know that such questions might occur on any exam. If you don't use multiple choice exams then this is easier than trying to find proper statistical measures.

A variation on the above is to have the subtle questions count for less than the "mainline" questions. That way the bulk of the grade is determined by overall understanding and the subtle questions matter only at the margins, which I'd guess is your intent in any case.

---

I note that the comment of user ctrl-alt-delor given to your question (testing language usage vs content) is also an important consideration. You can work around it on an individual basis in some situations as long as you are sensitive to it. But if your overall environment doesn't permit individual adjustments or mooting a question, you have to be more careful, likely avoiding the subtleties.

Answer 4

The more subtle questions might hinge on a distinction of singular vs plural or other 'wording' aspects, so that those who have the strongest grasp of the material can differentiate themselves.

You're grading attention to wording details, and not comprehension of the material. Grading attention to wording would be appropriate if you were teaching reading comprehension, but it isn't specifically relevant to teaching computer science.

It would seem impolitic for me to say that I made that item difficult deliberately, but it is true.

If you ask a trick question, you're playing a trick on your students. Ok, so I used a word trick in my last sentence, but there's some truth to it. You aren't making the question difficult by requiring deep knowledge of the material, but by requiring subtle understanding of the question. A good trick question announces itself.

To give an example, as a TA in a programming class, I gave my students some very short code snippets and asked them what they did (more precisely, I asked what the code printed, or the final value of a variable). I started with easy things like x = 3; x = x + 1 and finished with subtle things that I only expected the most advanced students to grasp, like x = x++ (it was a C class). I warned them in advance that there would be trick questions. I don't have exam results, since I was only TAing and not doing the exams, but from reading the students' faces, I do think that the brightest students understood, and the slower ones didn't but kind of got the idea that there was a difficulty, which is what I was aiming for. Note how the subtlety was packed in the code and highlighted, not hidden under a word trick.

To test a deep understanding of the material, explore finer points of the test material. For example, put some concepts that were covered in the lecture together in a way that wasn't covered, and see if the students are able to relate them to the lecture material.

Instead I usually point back to the textbook or my lecture to say that I actually explained and even stressed the point.

What's your students' reaction here? If it's “ah, I see, I hadn't understood/remembered that part of the lecture”, you're doing it right. If it's “ah, I see, I hadn't understood the question that way”, you're doing it wrong.

I usually get a good spread of numerical grades, so experience says that I am doing this correctly.

I don't see how that follows. (After all, random grade assignment would get a good spread.)

I think your students would be somewhat justified to consider your grading unfair, since it isn't based on knowledge or comprehension of the material. Yes, attention to detail is a useful thing, but trick questions, not so much.

You are also heavily penalizing students to whom your course is in a foreign language, and possibly students with reading disabilities. The former may or may not be acceptable; the latter, if it is the case, isn't.

---

In a workplace subtleties will be vital, so it is not a manufactured situation to include them in my teaching, and I only use subtle distinctions in tests where they exist in the material, and are worth emphasizing.

I'll now put my teacher hat back onto the shelf where it was gathering dust, and put my engineer hat back on. I regularly read and write requirements documents. (A test question is a kind of requirements document for the answer.) Requirements documents should be precise, but they should not be subtle. If I notice a subtlety in a requirements document, I'm not going to blindly implement what the document says, I'll send it back for clarification. If I'm writing a requirements document, I pay attention to being clear, I'm not going to convey a requirement through something like a plural. If the plural is at all important, I'll at least highlight it with a word like “multiple widgets”, and probably “multiple simultaneous widgets” or “multiple concurrent widgets” or whatever it is I actually want.

---

How do you handle situations where students push back on the wording of test questions or answers?

Given your description of the tests, in this situation, I think you should change the way you word your tests.

Answer 5

Although other answers have mentioned aspects of testing language skills for non-native speakers and other reasons why language focussed questions cause problems I felt the case for students with alternate needs in Computer Science specifically needs addressing.

I have much experience in working with and teaching Computer Science to students with various alternate needs. In my experience students with dyslexia, dyspraxia and other specific learning disabilities are not uncommon in computer science classes. I could go even further and say they are quite common in computer science classes. I expect to find that as many as 20% of an undergraduate computer science class has these characteristics, even if they have not been diagnosed.

Dyslexia is a condition where language cognition is lagging behind other measures of intelligence and cognition. In other words these are very capable and bright students who have physical conditions that impair their processing of language. To discriminate against some of the best students in a class and artificially inhibit their progress because of a condition they were born with seems bade educational practice.

You should certainly reflect on your testing methodologies. Learning vocabulary and technical terms is an important part of learning but it has to be assessed in an appropriate and non-discriminatory way.