3
$\begingroup$

Last year I taught a programming course, and used an automatic grading system for grading the students' homework (in addition to other evaluation measuares). It worked quite well; the bottleneck was the need to write test-cases for each assignment.

This year, I would like to have the students themselves write the test-cases - both to save time, and to have them practice this important skill.

I thought of the following scheme:

  • In each of the 10 assignments, 10% of the students will be the "test set" and the other 90% will be the "train set".
  • The "test set" students will write test-cases - each student will write e.g. 10 test-cases.
  • The "train set" students will solve the assignment; their solutions will be tested using the test-cases of the test-set.

The problem with this scheme is that, if the solution of student A from the train-set fails on the test-case of student B from the test-set, it is not clear whether it's because an error in student A's solution or an error in student B's test-case. Of course I can look and see who is correct, but this requires a lot of work and undermines the idea of automatic grading.

How can I improve this scheme to allow automatic or almost-automatic grading of both homework assignments and test-cases?

Improve this question
$\endgroup$
2
  • $\begingroup$ I have been coaching a student, that has been given lab work with unit tests to pass. While I agree that this is a good idea, we found that we had to write our own tests. To do TDD: you have to keep asking what is the next test?; What test will lead to the simplest next step? I don't know if this can be predicted. If we make a different choice then the next choices start to diverge, event though we end up passing all the tests in the end. $\endgroup$
    – ctrl-alt-delor
    Feb 16, 2019 at 16:10
  • $\begingroup$ Why not have all students write testcases and award a grade for each one, based on how the other students who answer it get on? If it has no errors then a full grade, less for errors. Have a set period of time for the writing activity, then move to students solving them - by then there should be so many that each student gets a random selection to answer. $\endgroup$
    – Solar Mike
    Feb 16, 2019 at 20:02

3 Answers 3

Reset to default
4
$\begingroup$

Have you thought about having a solution that you create and test the students test cases against your solution? Then you have a stable platform for running the student defined tests.

Also run test cases which you prepare against the student's code. Now you have graded the two parts independently of one another.

Just out of curiosity, what did you use for auto-grading?

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ If you have created a working solution, then by definition you have a set of tests. (If you have not tested it, then any assertion that it is working, is just an opinion, and it is safe to ignore it. $\endgroup$
    – ctrl-alt-delor
    Feb 16, 2019 at 16:04
  • $\begingroup$ Regarding your question: I checked many platforms, but did not find anyone that allowed the students to submit an entire github repository as a solution. So I wrote one myself: github.com/erelsgl/badkan $\endgroup$
    – Erel Segal-Halevi
    Feb 16, 2019 at 17:23
3
$\begingroup$

If you run all implementations against all tests, then you can see anomalies. If you put results into a grid, then any failures should show up as a line: horizontal for implementations, vertical for tests. Any odd failures will still need investigating, but there should be few.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Interesting. Now that I think of it, I believe that there must be some research in the statistics community about such situation. Can be intereting to check. $\endgroup$
    – Erel Segal-Halevi
    Feb 16, 2019 at 17:22
2
$\begingroup$

Very interesting strategy. You are essentially putting the 10% test case-writers in the role of a sophisticated recruiting agent who is "interviewing" the rest of the 90% of the class for a job. It is probably not really fair, since the 10% are probably not sufficiently mature to identify the key concepts to be tested, but the class discussion (or perhaps outraged complaints?) from the consequences (or should I say carnage?) should be extremely "rich". I hope you are ready for an interesting challenge.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.