Automatic grading is great for saving time on marking programming tasks, so there's more time to focus on actual teaching, but I've noticed a few flaws. Since work is essentially marked based on passing unit tests, thoughtful design often gets ignored, and excessively complex solutions are considered just as good as elegant ones.
I recently read this article about how this is a problem on MITx—instead of this solution:
A student wrote this:
1 * lo + 0 * x + 0 * hi # too low 0 * lo + 1 * x + 0 * hi # in range 0 * lo + 0 * x + 1 * hi # too high
Automatic testing would find these equivalent, but clearly one is incomprehensible and the other is relatively logical and sound.
This is also recognised in Using Test-Driven Development in the Classroom:
Despite its classroom utility, an automatic grading strategy like the one embodied in the Curator also has a number of shortcomings:
- Students focus on output correctness first and foremost; all other considerations are a distant second at best (design, commenting, appropriate use of abstraction, testing one's own code, etc.). This is due to the fact that the most immediate feedback students receive is on output correctness [...]
Is there any way of evaluating code style and design automatically (or at least detecting very poor solutions)? I've thought about using some static code analysis to try and evaluate solutions (e.g. cyclomatic complexity, or tools like these), although I've not tried them yet. Are there any reliable solutions to this, or must it be done manually? Does static code analysis help in any way?