Grading currently either takes me a huge amount of time, or gets done in an extremely cursory way. It occurs to me that, if my students were able to submit code into some sort of autotester, they would find many of their errors before I review their code. This would mean that I would only have to look for clarity issues, not logic bugs.

I am interested in software that (a) isn't too hard to set up, (b) will allow my students to test whether their code is (at least) to spec, and (c) allows me to quickly cycle through the code from all of the students in a section to look for style issues.

I have looked far and wide, and have not yet found a reasonable system that accomplishes these three goals. Many universities seem to have robust systems, but they all appear to be home-grown, and unavailable to outsiders. Is anyone aware of such a system? I honestly don't even care if it costs money - I just need a solution that makes this task more reasonable.

  • $\begingroup$ Software recommendations are hard on SE, but I could see this being useful to this community. $\endgroup$ Commented May 23, 2017 at 14:22
  • 1
    $\begingroup$ very useful. This is a huge concern. $\endgroup$ Commented May 23, 2017 at 14:34
  • $\begingroup$ To note it, there's an SE for Software Recommendations. But, I'm not suggesting that this is off-topic since it seems like a helpful type of question. $\endgroup$
    – Nat
    Commented May 23, 2017 at 14:36
  • 1
    $\begingroup$ I see this as a GOOD question about tool and workflow with a splash of pedagogy. For that reason I'd find it more valuable to keep this question here on the CS Ed site rather than on SE for Software Recs. $\endgroup$
    – andytilia
    Commented May 23, 2017 at 15:11
  • 1
    $\begingroup$ I'm curious to hear more on this topic. Almost tempted to start a bounty. I came across this thread regarding unit testing student work: stackoverflow.com/questions/28862949/… I'd like to hear what else people implement to automate the process. I do love CodingBat, but it would not work offline nor would it work if I don't first set everything up on their system. $\endgroup$
    – Peter
    Commented May 30, 2017 at 4:47

8 Answers 8


For simple stuff I've used CodingBat. They've got an authoring system buried on the site. It's not terribly difficult to create your own questions, although the instructions are pretty wordy.

When you create a problem there are a set of fields to fill in. I'd attach a screenshot, but the network I'm on blocks imgur. I'll try to come back and add an image later.


This is the instructions that your students will read.


If you want your students to have the hint button, add some text


List as many test cases as you want. Separate each parameter that you want to send into the method you're testing with a comma. The expected answer comes at the end.

It looks like this for the method below. 123 goes into sumEvenDigits as n and then 2 is the expected answer.


code Starter code for your students.

public int sumEvenDigits(int n) {


tags Let's you put questions into categories, specify language, and set a difficulty.

  • $\begingroup$ I've looked for this before, but I never found it. In your experience, practically speaking, how complex can an assignment be to be realistically tenable here? $\endgroup$
    – Ben I.
    Commented May 23, 2017 at 18:24
  • $\begingroup$ Works pretty well for warm up type stuff in an intro class. Not all that useful for labs once you get past single methods. I actually wound up writing my own because I couldn't do as much as I needed to with CodingBat. $\endgroup$
    – Ryan Nutt
    Commented May 23, 2017 at 18:30
  • $\begingroup$ Is your tool published? folds hands behind back, bats eyelashes innocently, whistles and looks down at feet $\endgroup$
    – Ben I.
    Commented May 23, 2017 at 18:35
  • $\begingroup$ There's a link on my profile. Don't want to post a link here. Seems a little spammy :) $\endgroup$
    – Ryan Nutt
    Commented May 23, 2017 at 18:38

I would recommend repl.it classroom It allows creation of a course and assignments within that course. Assignments can be auto-graded using unit testing (JUnit) or by providing inputs and comparing expected outputs against actual outputs. Students get feedback on what test cases passed and which did not. These can then be converted directly into student grades.

Here's a screenshot of one CodingBat problem:

repl.it classroom building an assignment

Repl.it also has the advantage of being completely cloud based -- no IDEs to install locally. Teachers and students both get free accounts. Java is just one of a dozen or more supported languages (Python, JavaScript, PHP, etc).

The developers are really responsive -- I've had several conversations with Amjad Masad on Twitter -- and they take feedback seriously.


I use VPL (Virtual Programming Lab) for Moodle. In my case, it is just used for the introductory PHP classes (since HTML and DB interactions aren't possible), but it supports a wide range of languages, including Java. If you have Moodle available in your school, it is just a matter of installing the plugin. Then you're able to create your own tasks and define test cases in plain-text files. Everything is available from the activity interface itself. If you know a bit of shell script, it is even possible to perform more checks on the code, including style checks (you can even deny a submission if it doesn't match your criteria).

Here is an example of a test cases file for a program that sums 2 values:

case = 1
input =1
output =3
case = 2
input =3
output =7
case = 3
input =5
output =11
case = 4
input =-1
output =-3
case = 5
input =-2
output =-1
case = 6
input =0
output =6
case = 7
input =7
output =7
case = 8
input =-10
output =-4
case = 9
input =10
output =1
case = 10
input =100
output =-100

VPL also supports automatic file comparison for plagiarism detection.


The textbook I use (Horstmann's Java for Everyone) has an autograder that the author has written. He has posted many of the exercises from his textbooks. You can find it at: Cay Horstmann's CodeCheck autograder

There is an authoring page that shows how to write your own problems. I've found this to be the easiest of all the tools I've tried when it comes to creating my own assignments.

The report page that is displayed when the students solve a problem includes the source code. Students may download a jar file with the results and submit that, but I just find it easier to have them print to pdf using their browser, and then upload that to our learning management system (Canvas). I can then browse through their code relatively quickly (although I still have to manually type in the grades while I'm doing that.)

Someone has written an Grading tool for CodeCheck and Canvas that downloads the uploaded jar files, allows you to comment on them locally and then uploads the grades back to Canvas. I found, however, that simply browsing the PDF printouts was more effective for me.

  • $\begingroup$ I've used CodeCheck myself for some years now, and it was particularly useful during the covid-19 pandemic. A particular feature is worth mentioning: it does not need users (students or instructors) to register! Everything works with randomly generated URLs. Nevertheless, you can also integrate it with an LMS through LTI. I've used it in Moodle and OpenEdX. $\endgroup$ Commented Mar 15 at 17:42

Disclaimer: I am one of the authors of this free service.

TL;DR: Use AutoGradr.com which has been battle tested in university classrooms

  • AutoGradr lets you create your own labs and projects
  • Labs: small, bite-sized questions to help students get familiar to a new concept. Students work in AutoGradr's web IDE and get instant feedback.
  • Projects: suitable for more complex questions where students can work in their own IDEs (Visual Studio, Eclipse etc.). They upload their solutions to AutoGradr and get instant feedback.
  • AutoGradr provides instant feedback to students allowing them to know their mistakes and make a better submission before it ever gets to you.
  • AutoGradr provides report on code quality to both students and instructors. Students can make newer attempts to improve their submission.
  • Instructors never have to write code to automate the grading.

AutoGradr projects have test cases that define what the program output must be. Test cases can either be visible or hidden.

Test cases can contain user input via stdin or files. Students can write to files and have them automatically tested.

Project on AutoGradr

AutoGradr's instant feedback tells student how their output differed from the expected output. They can make multiple attempts to fix any issues.

AutoGradr Feedback

Students and instructors can see report on code quality. Students can fix their code quality issues and make newer submissions.

AutoGradr Lint Report


That seem an important question. Here is my input. 'm using repl.it for a month, created ~60 assignments in 10 classrooms (java,c#). (Both basics, and also recursion, lined-lists and Queues. )

The idea of web-based-ide & auto-checking & status-report-to-teacher is fantastic ! It gets students very fast to code, and see results, also teacher knows immediately the situation of the class. Also some students find it hard to concentrate and listen to me, so they can go do tasks. It is very easy to learn to use repl both for teacher and student.

Teacher can create testbench and that saves a lot of time to students, so they can focus on their tasks, and get better testing to their code. Such testbench can include creating of various trees/arrays/linked lists , so students can focus on the method they need to write, and get a lot done automatically for them. It is easy to student to figure out what teacher wanted in a task, since she can see the testcases ! ( so it saves teacher communication time!).

We randomly divided a 30 HS-students-class in half, 15 with me and 15 with another teacher, my class using repl.it, had progressed much faster, compared to the others that used traditional ways.

**** repl.it is both great and at the same time, far from great ****

repl.it opened my eyes, now i want more from it. There are details that make the experience sub-optimal : -) test-cases need to be entered using many gui clicks, why can't there be one file,with many lines input/output (see VPL-Moodle) ?
-) Small tasks like renaming an assignment, accessing my solution, publishing, take way too many clicks, and between clicks we need to wait for web. -) no source control, if you by mistake clicked 'delete' for a test case it is gone forever.Same for changes to source code. -) hard to maintain assignments/classrooms over time, in collaboration with other teachers. Guess there should be text files (under wiki/github/jenkins etc.), that contains all assignment info (question text, start/test-bench code visible to students, testcases inputs and outputs , solution ) , that can be put to other teachers review and improve, like an open source or a wiki project. - hard to maintain common files (teacher utils) common to all assignments

I think that there is going to be some serious money in "web-based-ide & auto-checking & status-report-to-teacher" and would like to see some aggressive competition. Currently the only tool i know is repl.it.


We've been using Web-CAT. It's ok. It certainly motivates students to read error messages more closely.

EDIT: Here are three things that have bugged me about Web-CAT:

  • I remember having difficulty determining exactly where my unit tests were supposed to be uploaded; there's a user folder and an assignment folder, and one something's in the wrong place it's hard to just move it.
  • In three years, I still haven't found how to get it to accept "Type arrayname[]" as opposed to "Type[] arrayname"; students hate losing those two points.
  • Students sometimes have to click around a little to find report details, and the UI doesn't always make clear what's clickable.
  • 2
    $\begingroup$ Hi Chris! This is something of a stub answer. Would you mind editing it? Don't worry, the norms of answering on SE sites are not always obvious, and you're definitely not the first one to have a problem like this :) What can you tell us about Web-CAT? What are its strong points and its weaker points? $\endgroup$
    – Ben I.
    Commented Jun 30, 2017 at 1:31
  • $\begingroup$ Thanks for the note. I'll get the hang of this. :) 1) I remember having difficulty determining exactly where my unit tests were supposed to be uploaded; there's a user folder and an assignment folder, and one something's in the wrong place it's hard to just move it. 2) In three years, I still haven't found how to get it to accept "Type arrayname[]" as opposed to "Type[] arrayname"; students hate losing those two points. 3) Students sometimes have to click around a little to find report details, and the UI doesn't always make clear what's clickable. $\endgroup$
    – Chris M.
    Commented Jun 30, 2017 at 12:54
  • $\begingroup$ Oh, I meant in the answer itself :) Comments are considered temporary content, even if in practice they rarely get deleted. $\endgroup$
    – Ben I.
    Commented Jun 30, 2017 at 12:55
  • $\begingroup$ I only just now realized I can edit my answer! Thanks! $\endgroup$
    – Chris M.
    Commented Jun 30, 2017 at 13:05

Note: I'm one of the founders of CodeHS.

If you check out CodeHS, we have a system for writing autograders using Java. You can write unit test specific problems that test the output of a specific method or write generic autograders for any Java program if you want to test more complex things.

You can read more about them here: https://medium.com/read-write-code/these-are-the-autograders-you-ve-been-looking-for-bda0fa8fd8a

You can also see the documentation for our autograder framework here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.