How can I discourage poor design while using automatic grading of submissions?

Question

Automatic grading is great for saving time on marking programming tasks, so there's more time to focus on actual teaching, but I've noticed a few flaws. Since work is essentially marked based on passing unit tests, thoughtful design often gets ignored, and excessively complex solutions are considered just as good as elegant ones.

I recently read this article about how this is a problem on MITx—instead of this solution:

return max(min(hi,x),lo)

A student wrote this:

1 * lo + 0 * x + 0 * hi # too low
0 * lo + 1 * x + 0 * hi # in range
0 * lo + 0 * x + 1 * hi # too high

Automatic testing would find these equivalent, but clearly one is incomprehensible and the other is relatively logical and sound.

This is also recognised in Using Test-Driven Development in the Classroom:

Despite its classroom utility, an automatic grading strategy like the one embodied in the Curator also has a number of shortcomings:

• Students focus on output correctness first and foremost; all other considerations are a distant second at best (design, commenting, appropriate use of abstraction, testing one's own code, etc.). This is due to the fact that the most immediate feedback students receive is on output correctness [...]

Is there any way of evaluating code style and design automatically (or at least detecting very poor solutions)? I've thought about using some static code analysis to try and evaluate solutions (e.g. cyclomatic complexity, or tools like these), although I've not tried them yet. Are there any reliable solutions to this, or must it be done manually? Does static code analysis help in any way?

Answer 1

I am a strong advocate for code interviews. I make principles of style into about 50% of a lab grade (though I do permit kids to go back and refactor after an interview for full credit.)

For some context, I discuss regularly that there are two audiences of code: computers, and people, and that these audiences have very different needs.

Since my high school class is structured with lab periods, I will grade lab 1 during lab 2 work times (and so on). This gives me class time to get my interviews done. At the start, the code needs a lot of work. However, by the time we have arrived at lab 3, the kids are mostly creating crystal clear code.

Don't drop that autograder! Even with code interviews, it is still tremendously useful. Having a good autograder means that, during the interview, I only need to look for style issues. I can assume that even the strangest of their code (and boy, is there ever strange code at the beginning!) produces the correct output. That permits the interview process to focus in, laser-like, on how well the student wrote for the second audience.

Answer 2

Harvard's grading policy for CS50 is worth looking into.

There are four components for the grade on problem sets (each of which involves submitting code). The overall grade is calculated as scope * (3 * correctness + 2 * design + 1 * style)

Scope: to what extent does the code implement the features required by the specification?

Correctness: to what extent is the code consistent with the specifications and free of bugs? This is done by the check50 autograder, and it's essentially unit testing.

Style: to what extent is your code readable (i.e., commented and indented with variables aptly named)? there's a component for formatting: I think in Harvard's case these marks are awarded by teaching assistants, but basic static analysis or linting might suffice.

Design: essentially, is this good code in terms of clarity, efficiency, logic, elegance - again, Harvard use TAs to award these grades, and it's hard to see a machine (or an inexperienced grader) being able to award these marks accurately any time soon. If you were determined to use automatic grading, I guess you could do something with run times for test data or the more sophisticated forms of static analysis. A compromise might be the use of peer-assessment and a detailed, criteria based rubric: peer assessment might have other benefits.

Answer 3

First let me say that I think it is a good idea to automate as much as possible, and that there are tools that will help this.

The mystery test

However my experience with professionals is that there will always be a few that try to defeat the checker, not maliciously (they just don't get it; they are minimising error-messages, not minimising errors). So full automation may not be possible.

Therefore:

Add a few unpublished tests: If pupils meet the specification the test will pass, but you don't tell them the test or the result until grades are awarded.

Style award

You don't need to spend a lot of time on the manual part. If the style is good then it should make sense immediately (top marks), or mostly make sense (medium), does that work? (zero marks).

Therefore:

Award a style grade 0,1 or 2 based on a quick look at the code.

Big-O

I have seen on codility where they automate everything. There last test is always to check big-O. They run the code on a big data set, to check that run time does not increase worse then expected. They always tell the candidate what big-O to aim for.

Therefore:

Always have a test to check big-O. Tell pupils upfront, that order to expect.

Answer 4

Adding static code analysis can be useful in determining if students have "over-engineered" the solution. While dynamic testing can show that the program produces the correct output, there are many examples of correctly working programs that I would not consider acceptable. Students can sometimes miss an important concept that you are attempting to assess, but work around their misunderstanding by adding additional code that wasn't necessary.

I have done some testing of this by looking at code complexity (cyclomatic/McCabe and Halstead) numbers, state space, and LOC. For these values to be useful as a guide, the instructor must know expected value ranges for typical solutions to a given problem. These can be found by using exemplar solutions, either created by the instructor, or even identified from student submissions.

In cases where correctly working submissions have been written "incorrectly", I will address the situation with the student and make sure they understand the concept that they appeared to dodge.

If anyone's interested, I discussed this some, and included an example of this problem, in a SIGCSE paper a few years ago. If you want a copy of the paper, just email me - raymond dot pettit at acu dot edu:

Raymond Pettit, John Homer, Roger Gee, Susan Mengel, and Adam Starbuck. 2015. An Empirical Study of Iterative Improvement in Programming Assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE '15). ACM, New York, NY, USA, 410-415. DOI: http://dx.doi.org/10.1145/2676723.2677279

Answer 5

I would recommend looking into a static code analyzer or linter for whichever language you are using. These are designed to examine code for any unintelligible code and to make sure that the code follows the best practices. For example, a code analyzer or linter may suggest that you limit your line length, keep your methods small, and reduce unnecessary complexity.

Now, while this sound very useful, this is something that you have to be careful with and configure very carefully, because some of the metrics that are used can trigger it when the code is actually fine. It may be good enough for your situation to simply give the code analyzer to your students and let them use it to optimize their code.

If you do use a static code analyzer or linter, I'd recommend setting the limits high for how much room the students have for mistakes, because the contortions to get around a strange reason their assignment was poorly graded could make things much worse.

From some quick research, it seems like the most widely used python linter is pylint. From a quick glance at the docs, it seems to fit your goals.

As a rubyist, I use rubocop which is both a linter and static code analyzer.

Answer 6

Using lint tools is the correct answer. But not the easy one.

A lot of people here have suggested (and correctly so) to use a static code analyzer or linter. However, these tools can overwhelm students, especially beginners, with a lot of information that they may not understand.

I suggest using an auto grading tool that incorporates lint reports in it.

As an example, here is a lint report that we provide automatically to students after they make a submission:

It gives students an overview on how they did using the star ratings. Additionally, it provides clear and actionable items on how to improve their code quality.

Answer 7

I don't think you can just by using an auto grader.

I use an auto grader to tell me if the code solves the problem it's supposed to solve. And I tell my students that any code that solves the given problem is correct. Some people write a 2 line solution, some write a 50 line solution. But I don't consider either one more correct than the other. Generally the shorter solution takes more mental work and the longer solution takes more physical work (typing).

But I will go back and look at the submissions. Since the auto grader has checked it, I don't have to worry about whether the code builds or gets the right answer. That takes out the time consuming, boring part for me and lets me focus on style and any tips that I can give.

I guess, especially early in the year, I don't want to penalize students for a working solution that might not be as clean or elegant as another student's. There's enough frustration in a first year course without worrying too much about style and efficiency.