# How to design a cloze test for a programming language?

I teach a course on C++ for 2nd year undergraduates in computer science. So far, the exams in my course contained mostly code-writing questions, such as "write a class that implements a binary tree and supports the following methods: ...", or "write unit-tests for a class that implements a complex number testing the following methods: ...". During the semester, the students have very similar homework assignments, so students who have worked the assignments on their own should get high grades on this exam.

The problem is that such an exam is very hard to read and grade. So this year I would like to try a differnet kind of exam, similar to a cloze test. The idea is to give them an implementation of a class (e.g. a binary tree), but with some important words or statements missing. They have to complete the missing words. For example: there is a destructor heading but its content is missing, or vice-versa: the content is there but the heading is missing. I plan to have about 100 missing words, each of which is worth 1% of the total grade.

My questions are:

• How can I design such a cloze-test such that it will be easy for students who have solved the homework assignments by themselves, but hard for students who have not?
• Is this cloze-test used anywhere else for a programming language? I will be happy to see working examples so that I can get design ideas.

## 2 Answers

I don't think that an entire Cloze test is practical, and in any case, it does not assess deep understanding well.

One problem with longer Cloze examples is that they force the student not only to understand the goal, but also your particular implementation, which will often feel quite foreign to them, even if they are strong students. A further problem with Cloze-style is that, the more blanks you leave, the more possible combinations of correct answers you leave.

I once had a student fill in a top portion, meant to initialize a variable prior to a loop, with a correct lambda statement that solved the overall problem, and then fill in the following loop fill-ins in such a way as to make sure the loop never entered. The solution was a good one, so it received full credit.

I use a mix of problem types on my exams. Short Cloze-style questions can address syntactic understanding and the ability to accomplish very basic operations, such as identifying three compatible statements to put in a for loop to look through an array.

Multiple choice questions are good at making sure that particular distinctions are being made. I use these to isolate what I call "cognitive traps".

Coding exercises are, indeed, hard to grade, but they are also necessary for testing whether everything can be pulled together.

If you need to bring down the grading time, you may want to consider keeping the examples short and making AP-style code rubrics, as they can make the grading far easier.

Let me suggest a deeper analysis of the situation. As you describe it, I think it is too simplistic. Or, if not, then too hard to communicate standards.

In particular, why should every "fill in" have the same value? Different parts of a program are more important than others. In Java, for example, missing a semicolon is almost never an issue when you use an IDE that does various sorts of completions. Spelling errors are very hard to make in Eclipse (and others) for example. And, as a very experienced programmer with many languages and paradigms under my belt, I do depend on these "creature comforts" in programming. Even this site has a spell corrector.

And if the fill ins have different weight, I think it will be hard to communicate that in a way that doesn't impede the student work. Certainly in needs to be transparent.

But let me suggest something similar, though it won't solve your "easy to grade problem". I question the validity of that in any case.

There is a "Fixer Upper Pedagogical Pattern" (google it) that describes a certain kind of exercise. It might be adapted for an exam, especially if it were earlier used as a homework assignment. The idea is that you write a fairly significant program that solves an interesting problem. This should be a few hundred lines of code, perhaps. You then break the program in a few places. Some of the errors you introduce are purely syntactic (and perhaps fixable by an IDE). Others are deeper and require more understanding of the program and the problem it solves. I would never use it with an error in which the wrong underlying algorithm were used, however. The program is called a "fixer upper" and the student task is to repair the program (and pass a test suite, perhaps).

I don't think that this sort of exercise will separate the superstars from the plodders, but rather the superstars will find it easy and the plodders will need to spend more time with it. But that is exactly what you want them to do.

Since you know where the program was broken it may not be an extremely difficult task to find where they have made changes and a diff scan will show you that in any case.

Caveat. I've never had to scale a course beyond about 50 students, and seldom beyond 30, so have no experience if you are closer to 80, or 800. But at those scales, unless you have a lot of help in managing the course, the educational system itself is broken and should be fixed. Students need individual feedback.

• Thanks for the comment regarding different difficiulty levels. Indeed I did not intend to check spelling mistakes - only substantial language elements (and indeed, the educational system is problematic - I have 240 students. I have TAs who give feedback during the semester, but grading the final exam is my duty). May 16 '20 at 19:43
• There is a way to get TAs involved in grading. Have a single mass grading session with yourself and some (all?) TAs. Each TA only looks at a single question but the same question for every student. If they have any doubt they ask you for advice. We used to do this in Calculus courses where there were many teachers and many TAs. You probably need a training session before the grading session where you also present the rubric for each question. With more TAs than questions you could have a pair responsible for a question and each reviews the grades of the other. May 16 '20 at 20:10