Using autograding (Github Classroom) - pitfalls and best practices?

Question

I'm currently as a TA developing a series of programming assignments for a bachelor course on machine learning. Since we expect the number of students taking this course to grow in the future, we're looking at autograding to see if it's useful for us. In particular, we're looking at using Github Classroom. I have a number of related questions about this. Feel free to answer all or only some of them.

1. Autograding leans heavily on unit tests to grade, and these unit tests are in the students' repositories. This means students can work on their code until it satisfies all the tests, which would result in a perfect grade. What kind of assignment setup/grading setups are possible that aren't so all or nothing?

2. Can you stop students from altering files in the repository that they're not supposed to alter? For example, if their starter code contains a file assignment.py and tests.py, can you enforce that they only change the assignment file?

3. Is it possible to apply hidden unit tests, that aren't visible in the students' starter code repository? For example, we might want them to implement a ML algorithm, then test it on a dataset they haven't seen yet, to see if it generalizes well. Also, it would enable us to have a tamper-proof unit test (see point 2).

4. What are general issues to watch out for when using autograding for (some of) the assignments in a course? Or Github Classroom-specific problems?

5. What are in general best practices for using autograding in a course?

Answer 1

In the auto grader I use, the following process is followed:

1. The students files are copied into a pristine work area
2. The "provided" file(s) are removed from the work area
3. If I intend to use MOSS, the remaining files are copied for later use
4. The "provided" file(s) are copied into the work directory
5. The result is built (if necessary)
6. Tests are executed and graded

Note that the "provided" file(s) copied in at step 4 may be a superset of the file(s) provided to the student in the assignment and typically contain other/additional test cases.

If the build fails because a student has modified a "provided" file, the students gets no points. If Java assignments, the provided file is often a interface, sometimes an abstract class. In C/C++ assignments, the provided file is often a header file.

If some test cases are provided and reused verbatim in the final run, those test cases only account for 60% of the final grade. Thus, the student can not receive a passing grade by just making those test cases pass.

Answer 2

1, 2, & 3

In most cases, I'm okay with them having the tests ahead of time. The only thing that concerns me a bit is that they can edit the tests, and GitHub would use those.

What I do, and it's typically only on larger weight assignments, is to download all of the repositories. I'll then copy either a clean copy of the test files or a new set of test files and run that.

Normally I'll put the clean files in the root of the projects I download and run something like copy ../test.py . & python test.py to copy the clean version and use that.