5
$\begingroup$

BACKGROUND: I teach a C++ course with 300 students where the grade is based on programming assignments. We use automatic tools to assess the assignment correctness (using many unit-tests), code quality (using clang-tidy), and to check plagiarism. In addition, the students present some of their solutions to the lecturer or TA in an online interview, and are graded by six quality criteria: organization, testing, correctness, efficiency, readability, and safety (I used recommendations from here for the rubric).

Last year, while I was interviewing students, I noticed a paradoxical phoenomenon: about half the students wrote perfect code (100% in all rubrics), and they all used very similar ideas - similar tricks, algorithms, etc. While the solutions were not entirely identical (so I cannot blame them for plagiarism), I was quite convinced that some of them shared their ideas with others. On the other hand, about half the students wrote mediocre code (60-70% in most rubrics), and their work was very different than others - I was quite convinced that they did all the work by themselves.

It turned out that I gave a lower grade to students who did all work by themselves, than to students who (probably) consulted other students and used their ideas. My grading scheme does not properly encourage students to do original work. Note that the course guidelines do not forbid students to consult each other. But, I still want to encourage students to make the best effort to solve the homework by themselves.

QUESTION: how can I design a grading scheme for programming exercises, which simultaneously encourage both quality and originality?

$\endgroup$
10
  • $\begingroup$ Give a bit more background. What is the student/staff ratio for such a large course? How many TAs/graders etc. are involved? I think that other questions you've ask suggest you have a near impossible task. $\endgroup$
    – Buffy
    Commented Jan 11, 2022 at 16:30
  • 1
    $\begingroup$ @Buffy students present their homework in "lab" hours. There are 15 lab hours per week, so 20 students per lab. Assuming 5 presentations per hour, each student gets to present once in 4 weeks, which is 3 times per term (so they can present 3 out of 5 assignments). Indeed, it is very challenging. I asked to have 13 students per lab, but it was not approved due to budget considerations. $\endgroup$ Commented Jan 11, 2022 at 17:34
  • 3
    $\begingroup$ I have a strong sympathy with this question -- I've on occasion disliked myself for the grades I gave $\endgroup$
    – Rushi
    Commented Jan 11, 2022 at 18:26
  • 2
    $\begingroup$ @VictorEijkhout Until two years ago I, too, encouraged group submissions. Then, due to COVID we had to do oral examinations. I found out that some students, who got 100% homework grade, could not write a single line of code. When I asked "how did you get 100% homework without writing code?" they said "my partner wrote all the code. My job was just to submit the solution to Moodle"... Since then, I switched to individual submission. $\endgroup$ Commented Jan 12, 2022 at 10:35
  • 1
    $\begingroup$ @ScottRowe We have a kind of apprenticeship at the 3rd year, when students do individual projects. The course I am talking about is 2nd year, it should give students basic tools for doing projects. $\endgroup$ Commented Jan 17, 2022 at 9:25

2 Answers 2

4
$\begingroup$

Let me suggest two things, but I don't know if they will solve the problem.

The first is that once you set the rubric you are stuck with it until you publicly replace it. You may be right that there was collaboration but your "feelings" can't be acted on without evidence. I think you understand that. If it only mentions quality and completeness then it is that which you must depend on. But, you could also require explanations of things as part of any assignment. In Java, we have JavaDocs for that where we describe the intent of any class and of important methods. They are harder to grade, but also harder to copy in any real way, since they use natural language.

A short exposition, say a page, of what an assignment has accomplished and how could be submitted along with each assignment. Ideally it could include the "why" of major decisions, such as algorithm choice. Students would be required to explain their work, not just provide it.

The second thing is to spell out explicitly (for the course, and perhaps for individual assignments) what sorts of collaboration are permitted and which are not. But to make this effective (hmmm, more effective) you need to get them to actually affirm that they will respect it in all details, both the letter and the intent. It is one thing to say to students "You must do this." and quite a different thing if the student says to you "I agree that I will do this."

But in any conversation (oral or written) about an honor code, you need to actually explain why the rules are in place. After all, in the real world, collaboration is expected and the quality of the product is paramount. In an educational environment, it is the learning that is paramount, not the product. Students don't always understand that you didn't ask them to do something because you want the result. After all, you could have produced it yourself much more easily. Emphasize that any restrictions are to help ensure their skill development.

And, of course, if you forbid them from communicating with one another they you almost certainly need to provide ways for them to communicate with you. My technique for this was a private mailing list that all staff and students were subscribed to. Students could ask questions (but not post solutions/code) to the list. The questions would get answered, perhaps by another student - in public. Everyone saw every question and every answer. This has the additional educational benefit that when one student asks a question it is likely that others do also, but hesitated to ask for whatever reason. No one needed to feel "stuck" and then tempted to break the rules.

Like many things, students need to be taught how to behave.


Note, of course, that the above doesn't address "originality" explicitly. If you want to encourage that, you may want to find a way to get that idea into your rubric. But you would need to be careful to define what you mean. Variable names taken from Pokemon probably isn't helpful. One way to make this work is to offer a few points in the rubric for "extensions" to the assignment. They would have to be explained in natural language at least, and, perhaps programmed. But a few points for suggesting some "novel" extension to the assignment might work and be hard to fake if arrived at collaboratively.

$\endgroup$
2
  • $\begingroup$ I do not want to forbid collaboration altogether. I just want to reward students who did a larger effort to solve the exercise on their own. The idea of "extensions" seems very useful. I can design an exercise with 80% "fixed part" (e.g. write a binary tree class) and 20% "variable part" (e.g. write a function that prints out the tree in a nice format). Since 'nice' format is subjective, I can expect to have more originality (of course, the variable part cannot be graded automatically, only during the interview). $\endgroup$ Commented Jan 12, 2022 at 10:59
  • $\begingroup$ "After all, in the real world, collaboration is expected and the quality of the product is paramount. In an educational environment, it is the learning that is paramount, not the product. Students don't always understand that you didn't ask them to do something because you want the result. After all, you could have produced it yourself much more easily." - I really wish StackOverflow would understand that. There are quite a few good quality questions which explain that OP can't use feature X and SO generally goes crazy about how dumb teachers are for forbidding using particular tools. $\endgroup$
    – Fureeish
    Commented Jan 12, 2022 at 12:02
3
$\begingroup$

First, let me congratulate you on a really thorough grading procedure for such a large class. I am impressed! I think that you have created an excellent balancing act among the realities of grading code among a large population, and I'm sure that your students benefit from it tremendously.

Now, you wrote:

about half the students wrote perfect code (100% in all rubrics), and they all used very similar ideas - similar tricks, algorithms, etc.

It reminds me very much of this:

All happy families are alike, but every unhappy family is unhappy in its own way

―Leo Tolstoy, Anna Karenina, 1878

And I think it's true. In any happy family, partners agree upon roles together. Partners largely respect and listen to one another. Children are taught how to be in the world in a manner forged from love. The path to familial harmony is relatively narrow compared to the infinity of ways that things can go terribly wrong.

Turning back to code, while it's true that there are an infinity of possible answers to almost any coding prompt, it is also true that there are very strong norms to good, clean code. Many of your students will absorb these norms, so it makes sense, even without any collusion, that you will see many highly similar answers among those who do well.

In addition, algorithms are far more proscribed than the contents of paragraphs. In the end, they must work, and they should be understandable to others as well.

Once you take into account readability, there are only so many reasonable approaches to finding the maximum element in an array, memoizing a function, or performing a DFS on a tree.

So I guess I'd suggest that, at least from the evidence you wrote into the question, nothing is amiss, and you needn't worry. Instead, keep up your excellent grading procedure!

$\endgroup$
3
  • 1
    $\begingroup$ As an aside, I remember hearing a social scientist explaining that Tolstoy was wrong about this: that unhappy families all tend to have the same few problems, while happy families all find different ways of working them out. (Your answer still stands, despite that, of course!) $\endgroup$
    – gidds
    Commented Jan 11, 2022 at 22:35
  • $\begingroup$ @gidds The quote is certainly glib, but I think that, in truth, everyone and every situation is unique. You have to apply a grouping lens to see commonality, but truly, that's an artifact of our own brains. We really like to group things. In truth, I am quite sure there are exactly as many ways people are unhappy as there are unhappy people, and the same can be said for ways to be happy as well. Tolstoy and your professor were applying different lenses to map their understanding, but the map is not the territory. $\endgroup$
    – Ben I.
    Commented Jan 12, 2022 at 5:16
  • 2
    $\begingroup$ Thanks for the feedback. I understand that there is a limited number of perfect solutions. But I still feel that the current grading scheme does not do much to encourage originality and self-effort. In fact, it encourages students to just ask the top students in class to share their ideas with the group. I am trying to tweak the scheme a bit. $\endgroup$ Commented Jan 12, 2022 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.