How can I speed up the construction of high quality testing materials?

Question

Giving the same assessments from year to year causes fairly obvious problems. Cheating becomes more and more of a potential issue. Basic testing security dictates that we must create new assessments.

However, generating new material has a few problems of its own:

1. Quality goes down, because material is less vetted. There may be unclear instructions, or even outright errors.
2. Creating this new material just seems to take a really long time.

It is not unusual for the creation of a single test or quiz to take me 3-4 hours to design. I have to look over the curricular material and brainstorm problems that can get to the heart of the core issues. I then have to design the problems, test everything in an IDE (to make sure that there are, at least, no typos in the code examples), create grading rubrics and processes, and finish up with final formatting.

I tend to use a mix of multiple choice, free-response (coding by hand), and short answer questions The first and second of these are what students will find on the AP Computer Science A test, so I need to give students practice with these kinds of activities. The short answer questions are an easy way to check for understanding of some particular little point.

I have found that, between my various classes, I give an average of one quiz or test per week. If we assumed a 40 hour work-week (ha!), then designing each assessment fresh would be a commitment of 7.5%-10% of my entire work time. That doesn't count the time to administer or grade the thing, this is just for the design aspect.

In reality, then, I mostly don't do this, and mostly give assessments from prior years. I want to change this practice, but I don't see how I realistically can.

Are there strategies to speed up the design of written assessments without losing quality?

Note: I don't want to give fewer assessments, because labs are suspect, and because of the tremendous cognitive benefits of testing. Instead, I would ideally like to be able to design a high-quality test or quiz in under an hour. This amount of time would make the entire enterprise more tenable.

For the benefit of future visitors, I will also include my own answer, though given my times, it will (obviously) only be a partial answer.

Answer 1

I have a few suggestions based on my experience this year. I have one prep over two blocks, which meant that for every exam I wrote, I wrote two versions. Moreover, because I take a standards-based approach, students end up retaking the exam but a modified version at that. For some exams, I ended up with 4 versions. Here's how I sped up my own process:

Focus on proficiency. Not every questions needs to "trick" students. (Indeed, "tricking" should never be the goal of an assessment: it should be an accurate measure of what a student knows and can do at a particular moment in time.) If I believe proficiency is 70%, then 70% of the points should be relatively straightforward questions that all students should earn. Maybe another 20% consists of slightly more challenging questions, saving the final 10% for the questions that might focus on the most subtle of details. By focusing on what all students should be able to do, I can worry less after the "craftiness" of my questions and get right to the heart of the matter.

Write questions designed with substitution in mind. Let's say I'm asking questions to test a student's knowledge of basic Linux commands since it's necessary for our IDE. Whether I ask about ls, cd, mkdir, touch, etc. should be irrelevant; they are all fair game. Thus, one approach is to tell students to review from a list of terms, and come up with a potential set of 4-5 answers. Then, from class to class, you can simply change what term you are asking about. The answers don't change, but the questions do. No student will know in advance which terms will be asked about just that they will be asked about.

Keep the method, but change the input. I recently assessed students on Python, and I wanted them to be comfortable with two key concepts: the use of range() for iteration and the basics of string access and manipulation. I didn't have to change the instructions from class to class very much, but by changing the input, I gave the two classes unique exams. They knew to expect questions on strings and iterations, but until they are faced with the actual input, this bit of information does not necessarily help them answer the question.

I may have more ideas come to me as I read more responses, but here's my final, overarching thought:

EXAMS ARE NOT SURPRISE PARTIES. Students, assuming they have been present and attentive in class, should not be surprised by the content of your exams. Let them be occasions for students to show you what they can do. While we as educators might stress over the perfect question to ask about, say, an ArrayList, it should always come back to this: "How I can fairly and accurately assess what a student knows?" It is not, however, a chance for us to show how clever we are. Get to the point. Be clear; be fair. Let students prove how much they have mastered. Keep that mantra at the forefront, and test design can become simpler and faster.

---

ADDENDUM:

Learn LaTeX. I use LaTeX for the creation of all my exams. While the first couple exams took some time to get just right, now that I have written a large number using it, I cannot imagine using anything else. It takes care of the formatting, the indenting, the point scale, everything. I don't waste a lick of time worrying about how it will print or format, especially when I need to include code samples. The time you invest in learning it will speed up the time it takes to format your exams.

Answer 2

You don't necessarily have to rewrite the entire thing every year.

Most of the tests I give are 40 questions. What I try to do is add 10 or 15 each year to the bank that the test is pulled from. That way, each year there's some new stuff. But nobody ever sees the entire set, so questions from past years are new to them.

Tweaking what you have. For some questions, it's pretty easy to tweak an existing question.

for (int i=0; i<10; i+=2) {
    System.out.println(i * i);
}

You could pretty easily change the starting and ending values, how much i goes up or down each time, or what you want to print.

Speed up what you can. I like ExamView for building banks, but it's really clunky to actually enter questions. What I've been doing this year is type the questions in Markdown, exporting as either RTF or DocX, and importing that into ExamView. I can create questions way faster in Markdown and MathJax than I can when I have to click around in ExamView.

Rinse, repeat The other answers have really good ideas too, and no reason to repeat them here.

Answer 3

Here are a few strategies that I have found helpful:

1. Test every bit of code in an IDE or a live coding environment

   As much time as double-checking code takes, it still takes much less time than unanswerable questions that you ultimately can't count in the final grades.

2. Create the answer keys directly in the examination file.

   This one actually saves me a lot of time. If you use Microsoft Office, simply color the lettering of any answers white. When you print to a standard black toner laser printer, these answers are not printed at all. This practice also helps you gauge how much space the students will need for a free response question.

3. Create your rubrics for grading in the question space as well.

   Placing a rubric or grading guide into the exam (again, in white) both makes you reflect on the original question (thus helping to ensure quality) and makes it easier to grade later.

4. Build a question bank over time.

   This combines particularly well with numbers 2 and 3. It allows you to slowly recycle questions from years past, keeping keys and rubrics intact as you go.

Answer 4

I'm not a big supporter of your underlying concept here and it seems like you are overly dependent on exams. I prefer a teaching/grading system in which students don't feel so much at risk. It also has the benefit of making your core concern much less important. And even if you can come up wit questions "quickly" it is impossible to say they are really good questions without some verification.

Your first link is about courses with hundreds of students. I don't think it applies at most people's scale of things. The second link about testing leave a lot of it valid even if the "testing" isn't closely coupled to "grading". In other words, weekly quizzes could count for a tiny fraction of the overall assessment and still be valuable. In that situation, the criticality of getting a quiz "right" goes down dramatically. Who cares if a poorly worded question gets through if it counts 0.001%. You can also agree with your students that such questions will be dropped. In fact, any question in which a substantial fraction answers wrongly or doesn't answer at all is suspect. Another option for reducing risk is to simply drop some number of low grades for any student or allow re-testing (makeups) for poor answers. Dropping the lowest grade, for example, just recognizes that students aren't superhuman. This also reduces the need for the instructor to spend inordinate amounts of time on questions that are difficult/impossible to vet properly and for which the grades may have little to no correspondence with learning.

When I took Calculus 1 (back in the dark ages) the instructor gave a "pop quiz" nearly every day. Just one or two questions. We hated them (and often him as well), but they counted for very little overall and they encouraged us to review the previous lecture before the next one. He didn't take a lot of time preparing them (I'll guess), but he did have to review them - class size about 30. We didn't like him much but he did force us to learn (stick like, not carrot like). But other teachers were able to actually inspire us to learn without the stick.

So rather than "solving" the problem you state, I'd rather you find a way to simply make the problem go away altogether.

---

On the other hand, one positive thing I can say about your process is that you give students a lot of practice taking tests. If you want students to do something well, give them a lot of practice doing it. If you want them to do well on tests (including standardized tests) give them a lot of practice doing tests.

Answer 5

There are a few steps involved in the creation of test questions, and some of them can be time-sinks that are not amenable to any optimizations. The typing of the questions, answers, and rubric, for example depend on your typing speed, and that's not subject to change with new techniques. Your choice of tools for creation of the questions might make a difference. Changing tools, however, is going to come with a learning curve to switch, which probably destroys any gains for a while. Of the steps involved I can only think of two that seem open to techniques for reducing the time spent. Leverage scale and spiralize the writing process.

Scale

Many of the questions on an exam are going to be simple tests of knowledge. Such questions are in the dime-a-dozen category, using relatively small amounts of time to create, and needing very limited vetting. The original question seems focused on the questions that involve the student writing a code block, or program, as an answer. Faster generation of such question can be accomplished by using scale: scale the questions up or down, and the economy of scale concept.

• Keep the questions atomic. The less complex the question, the faster you can create it and, in your own mind, vet it. So reducing the scope of a question increases the speed of making it.

• Make questions that have larger answers. The ratio of time spent imagining and designing a problem to what it tests seems to geometric. A little extra time working on a scenario can make the question cover a lot more concepts. You can have fewer questions that require more work from the students, and cover more concepts from the course, while still testing all the points the exam should assess.

• Make the question bank in one pass, not over time. (This is an investment in the future. More time spent today less time tomorrow.) When you are trying to create a question that will cover a concept your mind is focused on that point. You probably have several ideas and spend time choosing which one to use. Don't choose one, choose several (six for a reason explained below). When you focus on the next concept your mind will switch gears and you spend time getting into that one and most of what's in your mind now will be lost. This uses the time spent focusing on the concept in question once rather than doing it again for each test.

Spiralize

Vetting of the questions seems to be up to the students. From the question's wording I am left with the impression that you consider a question vetted after it's been used on an exam, and was understood by the students well enough to answer the way you expected them to. That leaves me wondering what happens when a question does not pass the test. I'm going to guess that the problem of disconnect between what you wanted and what you got, when it happens, is connected to the same issue programmers solve by using naming conventions: familiarity. Solve that problem by creating the tests in a spiral pattern. If you use the last bullet above, you can even make it a double spiral.

This is a good place to use paper over digital, especially if the exams are issued on paper. Write down each problem you create in basic details. Using enough for you to recall it later without "fleshing it out" until later. Once written, so you won't forget it, move on to the next question. Having recorded all your ideas for question, return to the first and begin adding details, and cleaning up stray elements. What you "thought" the first time through is now muddied with everything you thought about since, and the question may be less clear than it was at first. Again, go through the questions, doing clean up, but not answers, yet.

Once all the question are "polished", you can return to the first one and begin to create your answers. If the questions still need polishing, certainly do so now. This pass through you are in answering mode, and more resemble the student's approach to the question. It's not new material for you, but it shouldn't be new to them either. If you can answer the question in the same way, now, that you'd expect the students to, then it's probably vetted enough to add to the question bank.

---

Why 6 questions?

Most schools cover a range of student years. Groups of four seems to be quite common. If a question is used from year to year it is possible that last year's students can provide answers to this year's students. With multiple questions available this can be reduced, but not eliminated completely, unless there are more questions than years in the school's coverage. Giving the same, exactly, test again later creates the possibility that the previous version is archived on the Internet somewhere - people will do almost anything, and students are no exception. Randomizing which question in the question bank to include on a test makes each test unique and any archived test will have limited value for the current one. If the bank has 6 versions available and you disqualify the last 2 used and the school does have 4 years worth of students, then the randomly selected question will have only a 25% chance of being seen by anyone currently in the student body, and the value of retaining previous answers goes way down for past and future students.

---

Bonus technique: Serendipity

Your mind is usually most focused on a concept when you are trying to explain it to students and answering their questions. Often you will have sudden inspirations of how to use, explain, or demonstrate the current concept. Such inspirations make good leads into great questions on an exam. Save them! In another answer¹ on note taking another user suggested the "Hipster PDA" for the students. It makes a great note taking tool for instructors as well. Use your own Hipster PDA to take notes of such inspirations when they occur and you can greatly reduce the time spent on "inventing" the problem for a question later. You cannot, unfortunately, plan to have inspirations, but you most certainly can plan to capitalize on them when they arrive.

1 Thanks to @Buffy for the Hipster PDA concept.