How to design a cloze test for a programming language?

Question

I teach a course on C++ for 2nd year undergraduates in computer science. So far, the exams in my course contained mostly code-writing questions, such as "write a class that implements a binary tree and supports the following methods: ...", or "write unit-tests for a class that implements a complex number testing the following methods: ...". During the semester, the students have very similar homework assignments, so students who have worked the assignments on their own should get high grades on this exam.

The problem is that such an exam is very hard to read and grade. So this year I would like to try a differnet kind of exam, similar to a cloze test. The idea is to give them an implementation of a class (e.g. a binary tree), but with some important words or statements missing. They have to complete the missing words. For example: there is a destructor heading but its content is missing, or vice-versa: the content is there but the heading is missing. I plan to have about 100 missing words, each of which is worth 1% of the total grade.

My questions are:

• How can I design such a cloze-test such that it will be easy for students who have solved the homework assignments by themselves, but hard for students who have not?
• Is this cloze-test used anywhere else for a programming language? I will be happy to see working examples so that I can get design ideas.

Answer 1

I don't think that an entire Cloze test is practical, and in any case, it does not assess deep understanding well.

One problem with longer Cloze examples is that they force the student not only to understand the goal, but also your particular implementation, which will often feel quite foreign to them, even if they are strong students. A further problem with Cloze-style is that, the more blanks you leave, the more possible combinations of correct answers you leave.

I once had a student fill in a top portion, meant to initialize a variable prior to a loop, with a correct lambda statement that solved the overall problem, and then fill in the following loop fill-ins in such a way as to make sure the loop never entered. The solution was a good one, so it received full credit.

I use a mix of problem types on my exams. Short Cloze-style questions can address syntactic understanding and the ability to accomplish very basic operations, such as identifying three compatible statements to put in a for loop to look through an array.

Multiple choice questions are good at making sure that particular distinctions are being made. I use these to isolate what I call "cognitive traps".

Coding exercises are, indeed, hard to grade, but they are also necessary for testing whether everything can be pulled together.

If you need to bring down the grading time, you may want to consider keeping the examples short and making AP-style code rubrics, as they can make the grading far easier.

Answer 2

Let me suggest a deeper analysis of the situation. As you describe it, I think it is too simplistic. Or, if not, then too hard to communicate standards.

In particular, why should every "fill in" have the same value? Different parts of a program are more important than others. In Java, for example, missing a semicolon is almost never an issue when you use an IDE that does various sorts of completions. Spelling errors are very hard to make in Eclipse (and others) for example. And, as a very experienced programmer with many languages and paradigms under my belt, I do depend on these "creature comforts" in programming. Even this site has a spell corrector.

And if the fill ins have different weight, I think it will be hard to communicate that in a way that doesn't impede the student work. Certainly in needs to be transparent.

But let me suggest something similar, though it won't solve your "easy to grade problem". I question the validity of that in any case.

There is a "Fixer Upper Pedagogical Pattern" (google it) that describes a certain kind of exercise. It might be adapted for an exam, especially if it were earlier used as a homework assignment. The idea is that you write a fairly significant program that solves an interesting problem. This should be a few hundred lines of code, perhaps. You then break the program in a few places. Some of the errors you introduce are purely syntactic (and perhaps fixable by an IDE). Others are deeper and require more understanding of the program and the problem it solves. I would never use it with an error in which the wrong underlying algorithm were used, however. The program is called a "fixer upper" and the student task is to repair the program (and pass a test suite, perhaps).

I don't think that this sort of exercise will separate the superstars from the plodders, but rather the superstars will find it easy and the plodders will need to spend more time with it. But that is exactly what you want them to do.

Since you know where the program was broken it may not be an extremely difficult task to find where they have made changes and a diff scan will show you that in any case.

Caveat. I've never had to scale a course beyond about 50 students, and seldom beyond 30, so have no experience if you are closer to 80, or 800. But at those scales, unless you have a lot of help in managing the course, the educational system itself is broken and should be fixed. Students need individual feedback.

Answer 3

Ben I. made a good point when he said " A further problem with Cloze-style is that, the more blanks you leave, the more possible combinations of correct answers you leave."

A reliable Cloze test, to be easily gradable, should ensure that every "blank" can be graded independently of each other. Ben I.'s example of a student who solved the entire problem with a Lambda statement in the first blank is a good example. For a non-computer science example of a "bad" Cloze test, consider the following "vocabulary" or "English language" Cloze test:

I gave the letter to ______. ______ took it to the ______ .

Now, a student might answer as follows:

I gave the letter to mom. She took it to the post office .

Another student might say:

I gave the letter to the post office. A postal worker took it to the mail sorting facility .

I'm sure you can think of a few more. What about:

I gave the letter to the police officer. She took it to the evidence room .

I gave the letter to General Vicks. His lieutenant took it to the war room .

I gave the letter to General Vicks's lieutenant. He took it to the general .

I gave the letter to the great brain-computer of Centaurus V. One of the brain-computer's robotic sentries took it to the Andromeda Galaxy .

All of these answers are reasonable, and all of them display inter-dependencies between the blanks. If the second blank includes a name or description of a man, it would probably be wrong for the second blank to consist of "she" or "it"!

One way you can accomplish this restriction is with an answer bank. For example, your students might be able to choose the correct answers from the following:

• i
• i++
• ++i
• i+f
• i<f
• i>f
• i>=f
• item.Focus()
• i.Focus()
• i.Focus(item)
• item.Location.Transition()
• f
• firstItem
• secondItem
• item.GetDiscriminant()
• i<3
• f==i
• i>r

Some of these answer choices might be red herrings, of course.

A second way to do this would be to not provide an answer bank, but tell students what they must put in each blank. For example,

for(int i=0; _____ ; i++) //write a boolean expression that ensures that the loop continues to run if i matches the loop invariant as follows....

A student who dropped in a complex Lambda that embedded a loop solving the entire purpose of the function the first time it is called and then exiting the function could then be marked wrong as although the function would work, the student would not have followed directions.

Answer 4

Like you, I teach C++, and to a very similar audience. I find exams pretty pointless. I assign a bunch of homework exercises during the semester that ask them something very specific, but at the end of the semester they have to do a programming project. (I have several project ideas, to prevent them from doing something too simple.)

The problem with C++ is that every student settles on a different subset of the language that they are comfortable with (for instance: for-i loop, vs range-based loops, vs modern ranges or writing their own iterator subclass). It looks to me like a Cloze test would force them in the straight jacket of your preferred subset.

Note: my classes are between 30-60 students. Reading the project writeups & judging the code takes me & a co-teacher about 2 days each.