# How can I automate the grading of programming assignments?

As a TA, the usual workflow for grading programming assignments for an introductory course would be:

1. Some instructors used email as a submission mechanism (yes, really.)
2. So, search for submissions in email and download every submission
3. Organize it to make grading easier later on
4. Go into each directory, compile it. Realize it doesn't work.
5. If I email the student, I get a response 'Works on my computer'
6. Debug it and make it compile on my computer
7. Due to lack of test cases, every student has demonstrated their work with their own version of output on the console
8. Decipher their output and hopefully they get a good grade
9. Partial grading is hard because I have to dig through the code and see what parts are done and how far away they are from a working solution

All in all, it takes about 7-10 minutes per submission if I do things carefully. 100 students, 3 programming problems = 300 submissions. 10 mins per submission = 3000 minutes = 2 days of continuous grading. Impossible.

I have heard instructors make scripts, use some tools. Sometimes, they get super strict on the output of the program so they can grade it easily. Worse, they make problems easier so it's easier to grade.

How can I automate this efficiently?

• Shouldn't you never have to debug your students work? If it doesn't compile, it seems like that is non working code, and an incomplete assignment. – thesecretmaster Jun 28 '17 at 23:33
• If you take C++ code that runs on Windows, it may not necessarily run on MacOS if it has platform dependent code. Introductory students make these mistakes all the time. – tusharsoni Jun 28 '17 at 23:39
• I probably wouldn't worry about 5 or 6. I'd leave it with "it doesn't work, come see me if you want to talk." On labs I don't worry about partial credit. It either works or it doesn't. – Ryan Nutt Jun 29 '17 at 1:51
• My university program had linux servers available for use in labs and remotely via SSH. All of our code had to compile on a specific server (which we could not install packages on) and were submitted into a directory on said server. It made the acquiring of code and execution environment very simple to manage. – Brian R Jun 29 '17 at 21:11
• Generally, it's best not to put a summary of the answers in a question. If you think a summary answer would add value, feel free to answer your own question instead to allow others to vote on it and edit it separately to your question. – Aurora0001 Jun 30 '17 at 17:10

The degree of automation that you can achieve depends on how you want to grade. The most valuable type of grading (to a student) does not only cover whether the code works or not, but also offers suggestions as to how to make it better, including readability, design, and so no. And that you cannot totally automate.

You can (and should) automate basic setups, compilation, and tests, and generally remove as much grading-boilerplate as possible via automation. Todo so, the first step is to make sure that students will follow certain reproducibility guidelines. In my case,

• The instructions for assignments always include a specific test environment and test command, that must run. So for example, this could be "jdk 1.7 as installed in the lab" + "javac of your .java files". This allows me to ignore "works for me in my place" excuses -- unless it works in the test environment, it will not receive a passing grade. Note that this is generally not a the development environment; that may be eclipse or whichever other IDE they choose. I generally try to stick to the lowest common bare-metal standard.

• JUnit or other unit-testing frameworks are great to assert minimal performance. You can either request them to write their own tests (make sure to specify what these should cover), or provide them with pre-written tests (more effort, but hopefully more in-depth testing). Ryan Nutt's answer also offers good suggestions. Unit-testing is huge not only because it helps you to grade, but also because it makes students aware of their own errors; they will then fix these errors before submitting the assignments, which is a clear win-win.

• Mandating command-line friendly build tools (maven, scons, makefiles, scripts, ...) really helps to quickly pass the above tests.

• Students must use an online submission system (eg.: Moodle task) to upload their code.

Afterwards, in my grading setup,

• I download all submissions for the assignment, and each will be nicely tucked into files named after the uploading student. I then run a small shell script that extracts each submission into a single folder. After significant frustration, the script has been tweaked to deal with .7z, .rar, and .tgz archives instead of the requested .zip files.

• I write small per-assignment scripts that I can point at a submission and will

1. open my favorite IDE pointing at the submission
2. compare it side-by-side with previous submissions by the same author, assuming that this is an incremental project, using meld or a similar visual diff tool.
3. launch any corresponding unit tests to see if anything fails
• I then write a per-assignment check-list of items that I want to look at in each submission. This gets refined until I have graded 5 or so, when it sort of stabilizes.

• Finally, I get to grade the bulk of submissions, writing comments for students as I go, and often copying and pasting comments from previous submissions. When all are graded (and not before, because I sometimes have to backtrack or refine grading criteria), I copy-and-paste comments and grades to the online submission tool so that students can see them. The most frequent mistakes make the bulk of the next lecture.

So essentially I recommend standardizing as much as possible, and investing some time into per-assignment automation, in order to speed up actual grading.

• Thanks for your answer! Specifying a version for JDK definitely works for Java across platforms but other languages such as C++, C are platform dependent. Students write platform dependent code that works on their computer and doesn't on a different platform. – tusharsoni Jun 29 '17 at 17:43
• Your grading setup seems organized but still has a lot of manual work. We eventually ended up investing the time to automate all of that. Saved TAs and instructors a bunch of time – tusharsoni Jun 29 '17 at 17:44

Yikes, that doesn't sound like a fun way to grade. Sounds about like what I do with students that are competing in ACSL competitions. But that's only 6 or 7 students, 4 times a year. I can't imagine trying it with 100 students all year.

Here's what I'm doing, and what I've done.

This is what I'm doing now. Kids login to Canvas and it launches an LTI tool embedded in an assignment. They do their coding in Chrome and click the Test button when they're ready. They can keep testing until their happy with the grade. Grades get sent back to Canvas. I still have to transfer them to our actual gradebook.

I can then go back and download the submissions if I want to look at them for style or tips. Generally I try and do this with everybody on at least one lab per lab set.

What I've found is that students average about 11 submissions before they move on to the next lab. I only grade the last one, although I'll go back and look to see how they progressed.

Downside to this is that it works well for smaller labs, but not as much with larger projects. I can break down large projects into pieces that they can check, but eventually there has to be a full project turned in.

JUnit Tests

Before I moved online I would write a JUnit test for every lab, and usually I'd write two. One that I would include with the starter code and one, more in depth, that I would use to test.

Students could run the test code as many times as they wanted. Then, they'd submit their solutions and I'd run it against the other test code.

Biggest downside is it was pretty time consuming to write unit tests for every lab. I've gotten pretty quick, but it's still a time suck.

Walk around

Not ideal, but I can pick up a lot about students' code by walking around the room and watching them code.

I do this a lot with projects to get a feel for where the kids are on the project. By the time it's turned in I've already got a pretty good idea on what they've done.

Doesn't scale well though. Biggest class I've ever had was 34 students.

Sometimes I fall into this trap, especially with my second and third year students. I give them labs, they work on their labs, and I assume that they're done correctly if they've been working in class.

Only saving grace on this one is that labs are worth very little and tests make up a majority of their averages. So even with gimmie grades on labs, they can still get hammered on the tests.

Admittedly not a good solution, and one that I'm trying to work my way out of next year.

Eyeball It

Skim the code and see if you think it works. Same as the participation, this only really works with smaller assignments and when labs aren't worth all that much.

• I really like that you are using Online AutoGrader as an option. When we tried similar options, the issue was that students worked in a 'fake' environment i.e. their browser. We wanted to make sure they were learning to use the right tools such as their IDEs, learn debugging in them etc. – tusharsoni Jun 29 '17 at 17:46
• Indeed. It works better for small-ish lab type assignments, not as much for projects. On some projects though I'll break them into pieces and have them submit each piece online for testing. By the time they turn in their full project I've already got a decent idea of how their code is going to work. That said, I've found the opposite of the fake environment problem. Being a primitive IDE without any help makes the focus on the problem and helps learn the methods. – Ryan Nutt Jun 29 '17 at 17:53
• Which Canvas LTI tool do you use for the kids to write and test in Chrome? – iFrame Oct 20 '17 at 14:06
• It's one I wrote because I couldn't find something that I liked. There's a link in my profile. – Ryan Nutt Oct 20 '17 at 14:25

The key to efficient grading of assignments is to build easily gradable assignments in the first place. This does not imply easier problems, but, as unit testing makes code testable, this kind of discipline makes assignments gradable. Like many of the suggestions here, JUnit test cases are an important factor, but to use them you need to more completely specify the assignment (e.g. what functions/methods to write, etc.) For higher level grading, you can build a facade method that takes a big string as input.

There are two tools that I use for auto-grading of assignments: repl.it classroom and Web-CAT.

• Repl.it classroom is good for small assignments. It's a SaaS-based development environment that supports dozens of different languages. You can set up a course and put assignments in it complete with descriptions, test cases (unit-test or output-based), and progress reporting. Repl.it also supports LTI integration for most course management systems like Moodle, Blackboard, Canvas, etc. This is fantastic for homework.
• Web-CAT is good for large lab assignments. Students code in their own IDE (Eclipse, BlueJ, etc) and then submit the files for automatic grading. This uses JUnit heavily, although there are some plugins for output-based grading. Web-CAT is highly opinionated and wants students to write their own test cases and testing metrics are part of the students' grades. You can get access to Web-CAT for your institution through Stephen Edwards at Virginia Tech or you can download and install it on your own server.

(I am submitting 2 answers, because they are very different takes on the same problem, and merging them together would be of little value)

This is a full-automation setup I have used in a C++ data-structures subject.

• Install an automated answer judge (say, domjudge or mooshack). Setup username/password pairs for each of your students. Configure the judge to only accept answers in your chosen programming language.

• Provide your students with exact specifications of the compilation environment used by the judge. If you want full build reproducibility, run the judges on virtual machines and provide copies of these to your students; however, specifying the exact compiler version and OS is generally enough.

• Spend significant time preparing each problem statement, resource limits (time and memory), and the sample and validation inputs and outputs. (Great) examples of problems designed for automated judging can be found at the UVA site. Note that they come without validation inputs and outputs; I have not found open repositories with full questions, but there are several repositories of enthusiast users with working answers.

• Open the contest and watch the answers roll in. Be prepared to find & fix errors in the problem statement or input/output files unless you were very, very careful when writing them.

Fully-automatic validation leaves no room for error on either your part (statement, input-output specs) or the student's (off-by-a-comma is a wrong answer). For these reasons I recommend pairing up with someone who can double-check that statements are clear and that tries to solve problems before opening time.

To prepare a single statement, I generally

• write a draft of the statement, with sample input and output
• code a reference answer
• code an input-generator that can create valid but random inputs of configurable sizes
• generate a few nice-looking inputs and run them through the reference answer to get the validation input-output pairs

It is a lot of work (around 2hrs for each problem, QA included); but afterwards marginal grading cost is effectively 0.

Note that nothing prevents students from copying each other's answers (although you could run answers through a plagiarism-detection program). You should be aware that cheating unless conditions are tightly controlled is quite easy.

I have seen universities use linters and test suites to automatically check the correct state of the assignment so it can be submitted. Here is an example in Python or the student assignment helper programs in C, used for the book CS:App as another example of possible small programs you could include with assignments to make this easier to review submissions.

Some interesting projects for those working with block-based languages (such as Scratch, Snap! and Blockly):

For Scratch, check out Dr Scratch, which takes a rubric approach to evaluating how much 'computational thinking' is evidenced by a project. Whilst the analysis might seem a bit reductive, it can be used independently by learners and includes some useful guidance on how to progress. The developers describe their approach in this paper. Dr Scratch is built on Hairball, a Python module which does static analysis of Scratch projects.

A more conventional autograder, lambda, is being developed by Michael Ball for Snap! It's already integrated into UCB's Beauty and Joy of Computing MOOC, and I think there are plans to make this more widely available. Michael wrote about this for Hello World #3.

Chris Roffey has developed an autograder for Blockly used in the initial round of the TCS Oxford Computing Challenge programming challenge, although I don't think the code for this is shared publicly.

As a TA, the usual workflow for grading programming assignments for an introductory course would be:

1. Some instructors used email as a submission mechanism (yes, really.)

"Yes, really"? In the introductory course at my university we submitted our programs as printouts.

If you take C++ code that runs on Windows, it may not necessarily run on MacOS if it has platform dependent code. Introductory students make these mistakes all the time.

It seems to me that the mistake is using C++ for an introductory course. Complete beginners do not need help shooting themselves in the foot. The curriculum may be out of your control, but if you have the option then teaching the introductory course either in a strongly typed LISP (SML, Haskell, or something similar) or at least in a concise portable language (Python, Ruby, etc.) should make it easier for both the students and the graders.

I used stepik.org for my functional programming lab (in Java). It works like a charm. As for me, the only missing part is plagiarism detection. I believe you should it give it a go!

With stepik, you have the features of a basic LMS (learning management system). Basically you can publish an online course (with several chapters..); you can get "students" to join the course, publish assignments; if needed define time window to submit assignments.

Among assignments (quizzes, Fill in the blanks, ...), it is possible to set up code assignments. For the code assignment, most of popular programming languages are possible (including Java, Python, C/C++, SQL, ...).

In order to assess an assignment, you need to define a test case. Typically, inputs are read from the console (for instance using new Scanner(System.in).nextLine() in Java), then the student's solution output needs to match the expected output defined for the test case.

More details can be found in stepik assignement types

These are some popular tools that the community should look into for a reasonable grading setup:

• AutoGradr.com (Disclaimer: I'm one of the authors of this free tool)
• HackerRank.com
• domjudge.org
• mooshak.dcc.fc.up.pt
• So, did you develop your tool in response to the problems you were having at the time you wrote this question? – Ben I. May 14 at 18:38
• Yup! I was a TA and built this project to solve this problem. Now we have a community of hundreds of professors using this solution to improve their course. – tusharsoni May 15 at 19:08
• @tusharoni Does the contact email on the website currently go to you? (And would you object if I contacted you?) – Ben I. May 15 at 21:25
• Yup, go for it! – tusharsoni May 16 at 1:43

Please, take a look at Stepik. Actually I myself work at Stepik, but if you're looking for a free cloud-based solution, it might be really useful for you. Code challenges are tested using stdin->stdout and you can specify any test cases that will be used to check student's submission.