How can I design online tests and prevent cheating?

Question

As you may know, schools in many parts of the world will remain closed for a long time due to the coronavirus outbreak.

In Italy, the government has invited schools to provide remote teaching, and since many schools are subscribers of G Suite or Office 365, teachers are giving online lessons using web conferencing services (Google Hangouts Meet or Microsoft Teams).

Now, the main problem is with tests.

Regarding computer science, if we know that "it is spectacularly easy to cheat on computer science labs" (and in my tests students have to write code on paper), we can just imagine how simple is to cheat in an online test, whether it is an assignment to be submitted after several days, or a few minute quiz.

A different assignment for each student is not a solution, since lazy students could ask clever students to solve their assignment, so I think I am forced to abandon tests which require students to write their own code.

Maybe a better idea could be a quiz with automatically generated questions, like a code snippet and the request to determine the output for a given input, with different input values and variable names for each student.

Do you know any tool to create such questions? Or do you have any other suggestions?

Answer 1

I don't think there are any published tools to generate programming assignments, though there are articles that describe such tools (such as this one from a CUNY, or this one from Croatia, so one option is to try to contact those authors and find out whether they would share their results with you.)

I will state categorically that there is no way of guaranteeing that students will not cheat. As the old saying goes, "locks are to keep honest people honest." Putting road blocks in the way of cheating will help dissuade students from trying, both because it becomes more of a bother, and because it gives them a chance to remind themselves of the ethics involved.

The bit about ethics is important, actually, and it can be remarkably effective all on its own. Dan Ariely did an excellent set of studies about this, which he summarized hilariously in this TED talk. If you don't want to watch that, then you can read this blurb from his blog, though the talk is well worth the time and much more interesting.

"We used an honor code in an experiment at MIT in 2007. We asked a few hundred undergraduates to do some simple math problems but didn’t give them enough time to finish. We then asked them to score their own tests and tell us how well they had done, with payment of $1 for each question they reported getting right. The students were asked to shred their papers afterward—but our shredder didn’t really work, so we could see how they had done. Many cheated. We had a second group of students follow the same procedure but only after they first signed the honor code. Signing the honor code before the test eliminated cheating altogether.

The effectiveness of such a code doesn’t stem from signing it, though. It comes from being reminded about moral issues. If your students eventually stop thinking about the ethics code as they sign it, it will lose its power. If they keep reading and reflecting on the code, its effectiveness might increase.

I ask my own students to write their own version of an ethics code—not because I’m especially interested in their interpretations but because it helps to ensure thought."

Dan Ariely

So, one way of working around the COVID-19 problem is to combine both ethical thinking and a few roadblocks to help gently dissuade cheating:

1. Remind students about ethics up front, at the top of your exam. (I make them copy out, by hand, an honor statement. On a computerized exam, I would provide an image of the statement and make them type it.)
2. Schedule the exam in advance. Give the exam at a specific time, and a narrow window (say, 10 minutes) to begin the exam.
3. There is no need to make different exercises for each student. Make 3 programming problems, and assign each student to different problems using a rule. This decreases the chances that they will have a friend in the class with the same problem.
4. Require that the exam be finished within a reasonable time from the start time.

It's not foolproof. They could, for instance, have their parent sitting next to them, telling them the answers. But you can't do anything about such a scheme. What you can do is set up roadblocks and set the stage for ethical action on the part of your students. If Ariely's studies are any guide, you should get the large majority of students to submit honest work this way, and that is probably the best you can hope for.

Answer 2

I agree with Ben I. and Buffy. I'm planning to cancel the remaining exams in the semester (assuming my university ends up moving to online instruction) and replace them with projects. Many studies have shown that project-based learning is more effective for most learners, and especially for learners from underrepresented groups.

That said, if I needed to proctor an exam remotely, I'd write an exam that's taken online (e.g. using Jupyter Notebooks) and then have all my students share their screens with me. I might also ask them to keep their mics on and their volume off, so I could hear if there was talking going on, but students wouldn't be bothered by the noise coming from each others' settings. You could even just record the whole hour long thing as a video on your machine, and students would know that if they cheated there would be a recording that the professor could dig through in the future to catch them. I think that would be a strong enough disincentive to cheat. It doesn't help with silent texting, but it stops students from actually talking to anyone else who is physically present, or from googling when they should be working on the exam.

EDIT: I just learned about a software called "respondus lockdown browser" for exams. You get a two month free trial, which should get you through the end of the semester if deployed at the right time. It prevents students from opening any window other than the one you give them (e.g. to google) and it also films them using their webcam (e.g. to prevent checking the book or texting).

Answer 3

At my University, we had the chance to modify the syllabus, if needed, to remove or decrease the weight of "traditional" written tests.

While in most programming courses eliminating tests is not a problem, as most of us were already relying mostly on assignments, group works, oral presentations, and so on, there are some courses, were more traditional tests may eventually still make sense.

Most of them are assuming that these are open-book tests, so the questions involve some degree of understanding of the topics and not simply memorization.

Some of my colleagues are using a tool called Exam.net that is capable of recording the student's face and the exams are made in full screen, and whenever the student comes out full screen, the test may automatically be blocked, and the student has to ask the teacher to unlock it. Even Moodle has a similar feature that allows you to have a locked full-screen window, which can be used along with other tools such as Zoom to record the student's face. Some colleagues are also using Google Forms and Microsoft Forms along with Zoom for the same purpose. Some of these tools also have features to control the student's IP address.

The problem I see with these approaches is related to the fact that you are only seeing the student's face. You cannot see who else is in the room, who is typing, if the student has multiple screens, if the student is using TeamViewer of any other tool and someone else is answering on their behalf. So although it is possible to have some more control, in the end, these measures end up working more like some kind of a way to make us feel that we are doing the right thing to make the tests valid. In the end, I believe that there is not a 100% cheating-prof testing solution. If anyone knows of one, I certainly would also like to use it. :)

Answer 4

This isn't really an answer and requires long term thinking, but perhaps this is an opportunity to start to rethink how we evaluate students. I'm not a big fan of testing. Especially high stakes testing and testing dependent on memorization.

Perhaps there is just a better way to determine whether students have mastered the material. I depended much more on paired and small group projects for evaluation, combined with peer evaluation. This was toward the end of my teaching career after I'd learned some things about teaching and learning.

But pairs can work over the internet as can groups of up to about five or so students if you provide a rich set of communication tools. Also, pairs meeting face to face isn't as dangerous in the spread of disease as large groups (citation needed, I suppose). But people can also choose how to meet, on-line or in person, as long as you don't force them to meet face to face.

But, my base position is that if we can learn how to evaluate students without traditional testing it would be a step forward. The Pedagogical Patterns Project has some thoughts on all of that, of course.

Answer 5

Maybe a better idea could be a quiz with automatically generated questions, like a code snippet and the request to determine the output for a given input, with different input values and variable names for each student.

I actually have successfully implemented this (in part) for my first-semester programming tests this year. For example: Using the Blackboard LMS, there is a question type called "Calculated Formula" which supports a question with variable numbers, and a numerical answer based on a formula behind the scenes (supplied by the instructor). It then generates a number of iterations of each question with the variables randomly filled in according to supplied parameters. The list of question iterations is explicitly recorded in advance, so the instructor can spot-check that they are all reasonable. When the test is deployed each student gets a randomly selected variant from this list.

While this type of question is probably intended for other continuous science courses (physics, biology, etc.) I've managed to wrestle this into working for my basic computing class. For example, I've constructed questions that ask about the results from:

• Sequential assignment statements
• Arithmetic results (including integer division and modulus)
• Logical operators (in C++, where values are 0 or 1)
• Decision statements (if/else, switch)
• Looping statements (how many times will this iterate? what is the result of this accumulator?)
• Function calls (pass-by-value and reference)
• Array access and manipulations (accumulators)

There are ways in which this is limited and I'm fighting against the system. For example, the system has no built-in modulus operator, so I need to recreate that using the available division and floor operator. There's no looping operator, so I can vary array contents, but not the size of the array or the iterations in the accumulator loop. I have a question with a random sequential accumulator that uses the Gaussian formula to generate the result on the back-end. Ironically, it seems like more difficult questions have simpler formulas, and vice-versa.

But I've found this to be of limited use, because it doesn't test actual constructive programming, knowledge of terms, etc. These questions only constitute 25% of my Programming I tests, and I didn't see any particular way to use the same ideas in Programming II, for example. They also take quite a bit of creative work to construct, granted the system wasn't really designed to support them. But nonetheless I'm happy to have them as part of my repertoire.

Answer 6

I rather agree with Buffy, put simply, my argument is that a more collaborative approach does bear more fruit. Especially if the goal is to show understanding of concepts and a how to put them into practice to solve a common problem. Rather than focusing on an individual’s response, see how they interact and problem-solve in a team utilising all the different skill sets/ levels available in the group. Rather than putting the onus on the students to perform, it instead empowers them to take control of the assignment. Isn’t this what we want them to do in the workplace eventually? I have found that feedback from the group on how members contributed is fairly consistent with reality. Students were pretty honest and stern in their feedback! I work with 11 to 18 year olds, but I’ve tried the same model with adults with success.

Answer 7

I write my exams in a way that students need to know the concepts regardless of outside resources (other than someone else taking the exam for them). Consider doing away with questions that can be copied and pasted into an IDE and the answer is spit out.

A few examples may be to ask them to demonstrate short circuiting, or to identify an algorithm and determine if it is correct. I allow students full internet access, any notes and any IDE of their choice during my exams, but they are under a tight time limit. They are of course not allowed to collaborate and to the best of my ability, it looks like they are honoring that.

I agree with every answer that says to get rid of tests, but if you're stuck teaching APCSA, it is a disservice to not prepare students with tests.