What autograding software do folks use for Java code?

Question

Grading currently either takes me a huge amount of time, or gets done in an extremely cursory way. It occurs to me that, if my students were able to submit code into some sort of autotester, they would find many of their errors before I review their code. This would mean that I would only have to look for clarity issues, not logic bugs.

I am interested in software that (a) isn't too hard to set up, (b) will allow my students to test whether their code is (at least) to spec, and (c) allows me to quickly cycle through the code from all of the students in a section to look for style issues.

I have looked far and wide, and have not yet found a reasonable system that accomplishes these three goals. Many universities seem to have robust systems, but they all appear to be home-grown, and unavailable to outsiders. Is anyone aware of such a system? I honestly don't even care if it costs money - I just need a solution that makes this task more reasonable.

Answer 1

For simple stuff I've used CodingBat. They've got an authoring system buried on the site. It's not terribly difficult to create your own questions, although the instructions are pretty wordy.

When you create a problem there are a set of fields to fill in. I'd attach a screenshot, but the network I'm on blocks imgur. I'll try to come back and add an image later.

doc

This is the instructions that your students will read.

hint

If you want your students to have the hint button, add some text

cases

List as many test cases as you want. Separate each parameter that you want to send into the method you're testing with a comma. The expected answer comes at the end.

It looks like this for the method below. 123 goes into sumEvenDigits as n and then 2 is the expected answer.

123,2
2006,8

code Starter code for your students.

public int sumEvenDigits(int n) {

}

tags Let's you put questions into categories, specify language, and set a difficulty.

Answer 2

I would recommend repl.it classroom It allows creation of a course and assignments within that course. Assignments can be auto-graded using unit testing (JUnit) or by providing inputs and comparing expected outputs against actual outputs. Students get feedback on what test cases passed and which did not. These can then be converted directly into student grades.

Here's a screenshot of one CodingBat problem:

Repl.it also has the advantage of being completely cloud based -- no IDEs to install locally. Teachers and students both get free accounts. Java is just one of a dozen or more supported languages (Python, JavaScript, PHP, etc).

The developers are really responsive -- I've had several conversations with Amjad Masad on Twitter -- and they take feedback seriously.

Answer 3

I use VPL (Virtual Programming Lab) for Moodle. In my case, it is just used for the introductory PHP classes (since HTML and DB interactions aren't possible), but it supports a wide range of languages, including Java. If you have Moodle available in your school, it is just a matter of installing the plugin. Then you're able to create your own tasks and define test cases in plain-text files. Everything is available from the activity interface itself. If you know a bit of shell script, it is even possible to perform more checks on the code, including style checks (you can even deny a submission if it doesn't match your criteria).

Here is an example of a test cases file for a program that sums 2 values:

case = 1
input =1
2
output =3
case = 2
input =3
4
output =7
case = 3
input =5
6
output =11
case = 4
input =-1
-2
output =-3
case = 5
input =-2
1
output =-1
case = 6
input =0
6
output =6
case = 7
input =7
0
output =7
case = 8
input =-10
6
output =-4
case = 9
input =10
-9
output =1
case = 10
input =100
-200
output =-100

VPL also supports automatic file comparison for plagiarism detection.

Answer 4

The textbook I use (Horstmann's Java for Everyone) has an autograder that the author has written. He has posted many of the exercises from his textbooks. You can find it at: Cay Horstmann's CodeCheck autograder

There is an authoring page that shows how to write your own problems. I've found this to be the easiest of all the tools I've tried when it comes to creating my own assignments.

The report page that is displayed when the students solve a problem includes the source code. Students may download a jar file with the results and submit that, but I just find it easier to have them print to pdf using their browser, and then upload that to our learning management system (Canvas). I can then browse through their code relatively quickly (although I still have to manually type in the grades while I'm doing that.)

Someone has written an Grading tool for CodeCheck and Canvas that downloads the uploaded jar files, allows you to comment on them locally and then uploads the grades back to Canvas. I found, however, that simply browsing the PDF printouts was more effective for me.

Answer 5

Disclaimer: I am one of the authors of this free service.

TL;DR: Use AutoGradr.com which has been battle tested in university classrooms

---

• AutoGradr lets you create your own labs and projects
• Labs: small, bite-sized questions to help students get familiar to a new concept. Students work in AutoGradr's web IDE and get instant feedback.
• Projects: suitable for more complex questions where students can work in their own IDEs (Visual Studio, Eclipse etc.). They upload their solutions to AutoGradr and get instant feedback.
• AutoGradr provides instant feedback to students allowing them to know their mistakes and make a better submission before it ever gets to you.
• AutoGradr provides report on code quality to both students and instructors. Students can make newer attempts to improve their submission.
• Instructors never have to write code to automate the grading.

---

AutoGradr projects have test cases that define what the program output must be. Test cases can either be visible or hidden.

Test cases can contain user input via stdin or files. Students can write to files and have them automatically tested.

---

AutoGradr's instant feedback tells student how their output differed from the expected output. They can make multiple attempts to fix any issues.

---

Students and instructors can see report on code quality. Students can fix their code quality issues and make newer submissions.

Answer 6

That seem an important question. Here is my input. 'm using repl.it for a month, created ~60 assignments in 10 classrooms (java,c#). (Both basics, and also recursion, lined-lists and Queues. )

The idea of web-based-ide & auto-checking & status-report-to-teacher is fantastic ! It gets students very fast to code, and see results, also teacher knows immediately the situation of the class. Also some students find it hard to concentrate and listen to me, so they can go do tasks. It is very easy to learn to use repl both for teacher and student.

Teacher can create testbench and that saves a lot of time to students, so they can focus on their tasks, and get better testing to their code. Such testbench can include creating of various trees/arrays/linked lists , so students can focus on the method they need to write, and get a lot done automatically for them. It is easy to student to figure out what teacher wanted in a task, since she can see the testcases ! ( so it saves teacher communication time!).

We randomly divided a 30 HS-students-class in half, 15 with me and 15 with another teacher, my class using repl.it, had progressed much faster, compared to the others that used traditional ways.

**** repl.it is both great and at the same time, far from great ****

repl.it opened my eyes, now i want more from it. There are details that make the experience sub-optimal : -) test-cases need to be entered using many gui clicks, why can't there be one file,with many lines input/output (see VPL-Moodle) ?
-) Small tasks like renaming an assignment, accessing my solution, publishing, take way too many clicks, and between clicks we need to wait for web. -) no source control, if you by mistake clicked 'delete' for a test case it is gone forever.Same for changes to source code. -) hard to maintain assignments/classrooms over time, in collaboration with other teachers. Guess there should be text files (under wiki/github/jenkins etc.), that contains all assignment info (question text, start/test-bench code visible to students, testcases inputs and outputs , solution ) , that can be put to other teachers review and improve, like an open source or a wiki project. - hard to maintain common files (teacher utils) common to all assignments

I think that there is going to be some serious money in "web-based-ide & auto-checking & status-report-to-teacher" and would like to see some aggressive competition. Currently the only tool i know is repl.it.

Answer 7

We've been using Web-CAT. It's ok. It certainly motivates students to read error messages more closely.

EDIT: Here are three things that have bugged me about Web-CAT:

• I remember having difficulty determining exactly where my unit tests were supposed to be uploaded; there's a user folder and an assignment folder, and one something's in the wrong place it's hard to just move it.
• In three years, I still haven't found how to get it to accept "Type arrayname[]" as opposed to "Type[] arrayname"; students hate losing those two points.
• Students sometimes have to click around a little to find report details, and the UI doesn't always make clear what's clickable.

Answer 8

Note: I'm one of the founders of CodeHS.

If you check out CodeHS, we have a system for writing autograders using Java. You can write unit test specific problems that test the output of a specific method or write generic autograders for any Java program if you want to test more complex things.

You can read more about them here: https://medium.com/read-write-code/these-are-the-autograders-you-ve-been-looking-for-bda0fa8fd8a

You can also see the documentation for our autograder framework here