19
$\begingroup$

Grading currently either takes me a huge amount of time, or gets done in an extremely cursory way. It occurs to me that, if my students were able to submit code into some sort of autotester, they would find many of their errors before I review their code. This would mean that I would only have to look for clarity issues, not logic bugs.

I am interested in software that (a) isn't too hard to set up, (b) will allow my students to test whether their code is (at least) to spec, and (c) allows me to quickly cycle through the code from all of the students in a section to look for style issues.

I have looked far and wide, and have not yet found a reasonable system that accomplishes these three goals. Many universities seem to have robust systems, but they all appear to be home-grown, and unavailable to outsiders. Is anyone aware of such a system? I honestly don't even care if it costs money - I just need a solution that makes this task more reasonable.

$\endgroup$
  • $\begingroup$ Software recommendations are hard on SE, but I could see this being useful to this community. $\endgroup$ – called2voyage May 23 '17 at 14:22
  • 1
    $\begingroup$ very useful. This is a huge concern. $\endgroup$ – ncmathsadist May 23 '17 at 14:34
  • $\begingroup$ To note it, there's an SE for Software Recommendations. But, I'm not suggesting that this is off-topic since it seems like a helpful type of question. $\endgroup$ – Nat May 23 '17 at 14:36
  • 1
    $\begingroup$ I see this as a GOOD question about tool and workflow with a splash of pedagogy. For that reason I'd find it more valuable to keep this question here on the CS Ed site rather than on SE for Software Recs. $\endgroup$ – andytilia May 23 '17 at 15:11
  • 1
    $\begingroup$ I'm curious to hear more on this topic. Almost tempted to start a bounty. I came across this thread regarding unit testing student work: stackoverflow.com/questions/28862949/… I'd like to hear what else people implement to automate the process. I do love CodingBat, but it would not work offline nor would it work if I don't first set everything up on their system. $\endgroup$ – Peter May 30 '17 at 4:47
7
$\begingroup$

For simple stuff I've used CodingBat. They've got an authoring system buried on the site. It's not terribly difficult to create your own questions, although the instructions are pretty wordy.

When you create a problem there are a set of fields to fill in. I'd attach a screenshot, but the network I'm on blocks imgur. I'll try to come back and add an image later.

doc

This is the instructions that your students will read.

hint

If you want your students to have the hint button, add some text

cases

List as many test cases as you want. Separate each parameter that you want to send into the method you're testing with a comma. The expected answer comes at the end.

It looks like this for the method below. 123 goes into sumEvenDigits as n and then 2 is the expected answer.

123,2
2006,8

code Starter code for your students.

public int sumEvenDigits(int n) {

}

tags Let's you put questions into categories, specify language, and set a difficulty.

$\endgroup$
  • $\begingroup$ I've looked for this before, but I never found it. In your experience, practically speaking, how complex can an assignment be to be realistically tenable here? $\endgroup$ – Ben I. May 23 '17 at 18:24
  • $\begingroup$ Works pretty well for warm up type stuff in an intro class. Not all that useful for labs once you get past single methods. I actually wound up writing my own because I couldn't do as much as I needed to with CodingBat. $\endgroup$ – Ryan Nutt May 23 '17 at 18:30
  • $\begingroup$ Is your tool published? folds hands behind back, bats eyelashes innocently, whistles and looks down at feet $\endgroup$ – Ben I. May 23 '17 at 18:35
  • $\begingroup$ There's a link on my profile. Don't want to post a link here. Seems a little spammy :) $\endgroup$ – Ryan Nutt May 23 '17 at 18:38
5
$\begingroup$

I use VPL (Virtual Programming Lab) for Moodle. In my case, it is just used for the introductory PHP classes (since HTML and DB interactions aren't possible), but it supports a wide range of languages, including Java. If you have Moodle available in your school, it is just a matter of installing the plugin. Then you're able to create your own tasks and define test cases in plain-text files. Everything is available from the activity interface itself. If you know a bit of shell script, it is even possible to perform more checks on the code, including style checks (you can even deny a submission if it doesn't match your criteria).

Here is an example of a test cases file for a program that sums 2 values:

case = 1
input =1
2
output =3
case = 2
input =3
4
output =7
case = 3
input =5
6
output =11
case = 4
input =-1
-2
output =-3
case = 5
input =-2
1
output =-1
case = 6
input =0
6
output =6
case = 7
input =7
0
output =7
case = 8
input =-10
6
output =-4
case = 9
input =10
-9
output =1
case = 10
input =100
-200
output =-100

VPL also supports automatic file comparison for plagiarism detection.

$\endgroup$
5
$\begingroup$

I would recommend repl.it classroom It allows creation of a course and assignments within that course. Assignments can be auto-graded using unit testing (JUnit) or by providing inputs and comparing expected outputs against actual outputs. Students get feedback on what test cases passed and which did not. These can then be converted directly into student grades.

Here's a screenshot of one CodingBat problem:

repl.it classroom building an assignment

Repl.it also has the advantage of being completely cloud based -- no IDEs to install locally. Teachers and students both get free accounts. Java is just one of a dozen or more supported languages (Python, JavaScript, PHP, etc).

The developers are really responsive -- I've had several conversations with Amjad Masad on Twitter -- and they take feedback seriously.

$\endgroup$
4
$\begingroup$

The textbook I use (Horstmann's Java for Everyone) has an autograder that the author has written. He has posted many of the exercises from his textbooks. You can find it at: Cay Horstmann's CodeCheck autograder

There is an authoring page that shows how to write your own problems. I've found this to be the easiest of all the tools I've tried when it comes to creating my own assignments.

The report page that is displayed when the students solve a problem includes the source code. Students may download a jar file with the results and submit that, but I just find it easier to have them print to pdf using their browser, and then upload that to our learning management system (Canvas). I can then browse through their code relatively quickly (although I still have to manually type in the grades while I'm doing that.)

Someone has written an Grading tool for CodeCheck and Canvas that downloads the uploaded jar files, allows you to comment on them locally and then uploads the grades back to Canvas. I found, however, that simply browsing the PDF printouts was more effective for me.

$\endgroup$
2
$\begingroup$

Disclaimer: I am one of the authors of this free service.

TL;DR: Use AutoGradr.com which has been battle tested in university classrooms


  • AutoGradr lets you create your own labs and projects
  • Labs: small, bite-sized questions to help students get familiar to a new concept. Students work in AutoGradr's web IDE and get instant feedback.
  • Projects: suitable for more complex questions where students can work in their own IDEs (Visual Studio, Eclipse etc.). They upload their solutions to AutoGradr and get instant feedback.
  • AutoGradr provides instant feedback to students allowing them to know their mistakes and make a better submission before it ever gets to you.
  • AutoGradr provides report on code quality to both students and instructors. Students can make newer attempts to improve their submission.
  • Instructors never have to write code to automate the grading.

AutoGradr projects have test cases that define what the program output must be. Test cases can either be visible or hidden.

Test cases can contain user input via stdin or files. Students can write to files and have them automatically tested.

Project on AutoGradr


AutoGradr's instant feedback tells student how their output differed from the expected output. They can make multiple attempts to fix any issues.

AutoGradr Feedback


Students and instructors can see report on code quality. Students can fix their code quality issues and make newer submissions.

AutoGradr Lint Report

$\endgroup$
1
$\begingroup$

We've been using Web-CAT. It's ok. It certainly motivates students to read error messages more closely.

EDIT: Here are three things that have bugged me about Web-CAT:

  • I remember having difficulty determining exactly where my unit tests were supposed to be uploaded; there's a user folder and an assignment folder, and one something's in the wrong place it's hard to just move it.
  • In three years, I still haven't found how to get it to accept "Type arrayname[]" as opposed to "Type[] arrayname"; students hate losing those two points.
  • Students sometimes have to click around a little to find report details, and the UI doesn't always make clear what's clickable.
$\endgroup$
  • 2
    $\begingroup$ Hi Chris! This is something of a stub answer. Would you mind editing it? Don't worry, the norms of answering on SE sites are not always obvious, and you're definitely not the first one to have a problem like this :) What can you tell us about Web-CAT? What are its strong points and its weaker points? $\endgroup$ – Ben I. Jun 30 '17 at 1:31
  • $\begingroup$ Thanks for the note. I'll get the hang of this. :) 1) I remember having difficulty determining exactly where my unit tests were supposed to be uploaded; there's a user folder and an assignment folder, and one something's in the wrong place it's hard to just move it. 2) In three years, I still haven't found how to get it to accept "Type arrayname[]" as opposed to "Type[] arrayname"; students hate losing those two points. 3) Students sometimes have to click around a little to find report details, and the UI doesn't always make clear what's clickable. $\endgroup$ – Chris M. Jun 30 '17 at 12:54
  • $\begingroup$ Oh, I meant in the answer itself :) Comments are considered temporary content, even if in practice they rarely get deleted. $\endgroup$ – Ben I. Jun 30 '17 at 12:55
  • $\begingroup$ I only just now realized I can edit my answer! Thanks! $\endgroup$ – Chris M. Jun 30 '17 at 13:05
0
$\begingroup$

Note: I'm one of the founders of CodeHS.

If you check out CodeHS, we have a system for writing autograders using Java. You can write unit test specific problems that test the output of a specific method or write generic autograders for any Java program if you want to test more complex things.

You can read more about them here: https://medium.com/read-write-code/these-are-the-autograders-you-ve-been-looking-for-bda0fa8fd8a

You can also see the documentation for our autograder framework here

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.