I'm currently as a TA developing a series of programming assignments for a bachelor course on machine learning. Since we expect the number of students taking this course to grow in the future, we're looking at autograding to see if it's useful for us. In particular, we're looking at using Github Classroom. I have a number of related questions about this. Feel free to answer all or only some of them.
Autograding leans heavily on unit tests to grade, and these unit tests are in the students' repositories. This means students can work on their code until it satisfies all the tests, which would result in a perfect grade. What kind of assignment setup/grading setups are possible that aren't so all or nothing?
Can you stop students from altering files in the repository that they're not supposed to alter? For example, if their starter code contains a file
tests.py, can you enforce that they only change the assignment file?
Is it possible to apply hidden unit tests, that aren't visible in the students' starter code repository? For example, we might want them to implement a ML algorithm, then test it on a dataset they haven't seen yet, to see if it generalizes well. Also, it would enable us to have a tamper-proof unit test (see point 2).
What are general issues to watch out for when using autograding for (some of) the assignments in a course? Or Github Classroom-specific problems?
What are in general best practices for using autograding in a course?