Last year I taught a programming course, and used an automatic grading system for grading the students' homework (in addition to other evaluation measuares). It worked quite well; the bottleneck was the need to write test-cases for each assignment.
This year, I would like to have the students themselves write the test-cases - both to save time, and to have them practice this important skill.
I thought of the following scheme:
- In each of the 10 assignments, 10% of the students will be the "test set" and the other 90% will be the "train set".
- The "test set" students will write test-cases - each student will write e.g. 10 test-cases.
- The "train set" students will solve the assignment; their solutions will be tested using the test-cases of the test-set.
The problem with this scheme is that, if the solution of student A from the train-set fails on the test-case of student B from the test-set, it is not clear whether it's because an error in student A's solution or an error in student B's test-case. Of course I can look and see who is correct, but this requires a lot of work and undermines the idea of automatic grading.
How can I improve this scheme to allow automatic or almost-automatic grading of both homework assignments and test-cases?