27
$\begingroup$

A number of the assignments I've completed during the course of my undergraduate degree have been assessed partially using automated tests. These have never been available to students prior to the assignment submission, but I know this isn't the case at some universities. It would seem difficult to me to get less than 100% if all the test cases used for grading were released prior to the assignment deadline (provided the student gave enough time to the assignment).

In some cases, the feedback I've received has indicated that my submission failed a number of automated tests - but the test cases were never made available to students and I was left unsure as to what I had done wrong in my work and how I could improve next time.

In other cases, the test cases were made available after the assignment grades were released. I much preferred this approach, because I could see exactly where I'd gone wrong and the whole process felt fairer.

Should automated test cases be released and if so, at what point should they be released?

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ Depends if you are grading programming ability, or ability to guess what you are thinking. $\endgroup$
    – ctrl-alt-delor
    Commented Jun 25, 2017 at 17:31
  • 3
    $\begingroup$ In introductory courses at the U of Toronto, they generally incorporate a few key cases as examples in the assignment spec, and warn students that these aren't exhaustive — and then none are added that test behaviour not explicitly described in the spec. And, as @EllenSpertus says below, from the first course up there are assignments requiring test-writing too. $\endgroup$
    – Luke Sawczak
    Commented Jun 25, 2017 at 19:06
  • 2
    $\begingroup$ Isn't the problem of writing a test case map eliminated when instructors actually glance at the code? I'm not sure this justifies not releasing test cases, nor would I appreciate it if my grade was entirely determined by automated tests. $\endgroup$
    – Adam Williams
    Commented Jun 25, 2017 at 19:10
  • 1
    $\begingroup$ Releasing some simple test cases is a very good way for the students to check they understand what they are required to do. On the other hand, not releasing all the test cases used until after grading seems almost essential, otherwise the smart students will just "code to the test" rather than actually solve the problem assigned. That may get them good marks for the course, but it won't make them good real-world programmers! $\endgroup$
    – alephzero
    Commented Jun 25, 2017 at 21:13
  • 1
    $\begingroup$ In my experience some professors would make these tests available to students, some would describe them in technical detail in class the next day, and some would go through them individually during office hours. I don't think I ever had a professor keep the tests a secret. Have you tried going to the office of those who held out and asking? Certainly as an educator I'd probably follow all three approaches simultaneously, but they are by no means obligated to do so as finding the problem creates a very difficult, but optional, exercise for the student. $\endgroup$
    – Darren
    Commented Jun 26, 2017 at 18:53

9 Answers 9

Reset to default
50
$\begingroup$

My best practice would be to provide students with test cases and require them to submit additional test cases with their code. Then run everyone's test cases against everyone's code. Let the students know that they will get extra credit if their tests break anyone's code. That will put them in a mindset of creating tough tests and writing bullet-proof code, rather than stopping when the pass the provided tests.

Improve this answer
$\endgroup$
8
  • 6
    $\begingroup$ I really like this solution, seems like a great way to encourage students to think outside the box and write tests for situations that you may not have thought of initially. $\endgroup$
    – Adam Williams
    Commented Jun 25, 2017 at 18:26
  • 9
    $\begingroup$ But make sure the student-provided test cases should pass first. $\endgroup$
    – Criticizing Israel not allowed
    Commented Jun 26, 2017 at 5:32
  • $\begingroup$ A course at my Uni did this! There were two stages: At first we had to upload code that would be automatically tested against testcases and in the second stage we would upload testcases to run against the other students solutions. It was very fun to work against the testcases, but the second stage was to easy as most students had not beat many testcases from the first stage and also the objective wasn't clear (do we have to pass our own cases?). Also as @immibis said you should make sure that the test cases are correct by running them against a reference solution. $\endgroup$
    – KarlKastor
    Commented Jun 26, 2017 at 12:28
  • 1
    $\begingroup$ I feel like you would get too many badly written test cases that fail when they shouldn't. At the least have what you consider to be a very well written sample version of the code that you run the test cases against. I'd probably ignore every test case that fails a valid sample code. Sure someone may have written a test case that actually caught a bug in your sample 'working' implementation, but most of the time they won't and it's not worth the effort of identifying which tests are valid and which aren't. $\endgroup$
    – dsollen
    Commented Jun 26, 2017 at 17:47
  • 1
    $\begingroup$ @KarlKastor Are bad tests flushed out by testing them against a canonical implementation of the program? $\endgroup$
    – corsiKa
    Commented Jun 26, 2017 at 21:51
18
$\begingroup$

I've always released a number of test cases for the purposes of clarity. I do have to double- and triple-check that my unpublished test cases are nevertheless unambiguously specified in the documentation I provided.

However, I am a computer science teacher, not a mystery novelist. It would never occur to me to deduct points for test cases that are not provided in advance without also telling the student what the test case in question is at the moment of the deduction. That seems plum crazy to me; how would the students learn this way?

Improve this answer
$\endgroup$
1
  • $\begingroup$ "That seems plum crazy to me": It's one of a few things that has frustrated me with my department.. $\endgroup$
    – Adam Williams
    Commented Jun 25, 2017 at 18:25
10
$\begingroup$

TL;DR: If there's no special reason against it, release them before the assignments are done. If you'd like to keep them secret to the students while developing, I would still recommend releasing them after the assignment for the students to help them to evaluate their mistakes.

I think whether and when the test cases should be released depends on a few factors.

Not releasing the test cases

You say that the assignments were only partially assessed by the unit tests, so in my understanding this means that there is somebody having a look at the source code.

If (and only if) the assessment was fully automated and there were no human beings looking at the source code, you should not release the test cases before the assignment as otherwise it would be possible that somebody just created a program that has the required outputs for the test cases hard-coded and chooses the right one (using some kind of map or dictionary) depending on the input.

If you plan to put the same (auto-graded) assignment again on students in e.g. multiple courses, you might also want to avoid releasing the test cases after the assignment - otherwise students that have completed the assignment could be able to hand the required test cases through to the students that haven't yet completed the assignment.

If you however change the unit tests for an auto-graded system after releasing them (i.e. each course has it's own set of unit tests for the same task), then you could release them after the assignment.

Releasing the test cases before the assignment

I can imagine a situation where it's neccessary to release the test cases and to do this before the assignment: If you're doing test-driven development. In this case having all the unit tests available to everybody would be essential as otherwise the whole strategy wouldn't work.

Furthermore, the test cases could help students as they are examples of the program's in- and output if they have trouble understanding the specification (as specifications can benefit from examples).

I would even go further and would recommend you to release the test cases before the assignment in almost any case unless you have specific reasons against it.

Releasing it after the assignment

This could be an option if you want the students to figure out the algorithm without having examples (not sure when this is required, but maybe there are some sort of tasks that require that) but you want them to give a chance to review their mistakes after the assignment is done. I would always try to release them at the latest when the assignment is done.

Improve this answer
$\endgroup$
5
$\begingroup$

The only time I would not provide them, is if you are asking the students to derive test cases. You should not be testing to see if they can guess the test cases. Success criteria, should always be made available.

Exceptions, have most test cases available to pupils, but have one or two extra, but ensure that there is enough info in the question to derive it. One of the secret tests should be to test big O of the solution (as codility do on their on-line tests. They seem to specify the big-O for every question, but not the test).

Afterwards make it very clear where they lost grades (codility does not public the code for their tests, but does tell you what it tests, e.g. failed test of O(n²).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Not sure how I feel about the secret time complexity testing. I'd be okay with it if the spec explicitly asked for the most time-efficient algorithm to be implemented, but there's often a trade-off between space-complexity, time-complexity and hell .. readability. Should students be penalised for prioritising the "wrong" thing there? $\endgroup$
    – Adam Williams
    Commented Jun 25, 2017 at 18:29
  • $\begingroup$ @AdamWilliams yes specify big-O in the question. Unless it in implicit e.g. implement quick sort. And while I would always say optimise for readability, I would also say that minimising complexity is making it more readable. Having a loop within a loop when one loop will do is not cool, but having a loop follow a loop is ok. Having more code that me in the loop is also ok. Though we could ask, could that be moved out of the loop, it would not fail a big-O test. $\endgroup$
    – ctrl-alt-delor
    Commented Jun 25, 2017 at 18:38
  • 1
    $\begingroup$ "You should not be testing to see if they can guess the test cases." But you should be testing to see if they can write correct programs. "Not penalizing them for not deriving their own test cases" amounts to "not penalizing them for not dong the work properly" - unless the objective of the course is to produce graduates who can't manage "fizzbuzz" type questions in job interviews! $\endgroup$
    – alephzero
    Commented Jun 25, 2017 at 21:09
  • $\begingroup$ @alephzero you are infinity confusing, please cite all of your quotations. If we do not have a link to them how can we know what your point is. $\endgroup$
    – ctrl-alt-delor
    Commented Jun 26, 2017 at 17:19
5
$\begingroup$

Yes, Absolutely

Test cases are necessary to communicate what you expect the student's program to do.

Usually, instructors create 3-5 test cases and make at least 2 visible to the students. That way, students know what is expected but can't tailor their solution to just those test cases as they still need to accommodate for the hidden ones.

Test cases must include:

  • Exact stdout (output printed on the console)
  • Clear indication of what will be user input using stdin
  • Any files available to the student to read from (remember to tell the students what the name of the file is and the formatting of it)
  • Any files that the student is expected to write (tell them what path they should write to and how they should format their files)

Here's an example of a test case that is provided to students on AutoGradr: Test Cases on AutoGradr.com

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Very nice answer - I like the emphasis on clarity of the assignment itself. Welcome to Computer Science Educators! I hope we hear more from you in the future :) $\endgroup$
    – Ben I. ♦
    Commented Jun 26, 2017 at 17:42
4
$\begingroup$

Test cases are a great tool for students relatively new to the discipline. Since I teach CS50 AP, my students have access to the command-line tool check50, which they can run for nearly all the programs they write at the beginning of the year.

For my students who are new to the discipline, this helps them understand the need to read the problem specification thoroughly since each test case connects directly to something they are asked to implement. It's a relatively simple tool: if a test passes, they'll see a green :); if it fails, a red :(; if something unexpected, a yellow :|.

As the difficult and the size of the programs scale, students have to stop relying on the tool and instead have to test their code thoroughly based on a careful, thorough reading of exactly what they are being asked to do. Scaffolding in this way shifts the onus over the course of the year: they are given each test case in the beginning, but as the year progresses, it is more and more their responsibility to test their code for all possible inputs. This gradual transition works well. Here's how CS50 explains it when students reach the first assignment -- one which uses random numbers -- for which there is no check50:

When ready to check the correctness of your program officially with check50...well, you can’t. Reason being that the way the staff solution generates random numbers might in fact be different from your own, even though both do properly generate sets of random numbers. It’s up to you to determine that your program produces:

  • the correct number of pseudorandomly generated numbers,

  • each of which is greater than or equal to 0 and also less than (and not equal to) max, and

  • that if your program is run with the same seed value multiple times, the list of numbers it generates is identical from run to run.

Not having access to check50 for this problem is actually a good thing. It’s a bad idea to get into the habit of testing your code with check50 before testing it yourself. (And definitely don’t get into an even worse habit of only testing your code with check50!) Suffice it to say check50 doesn’t exist in the real world, so running your code with your own sample inputs, comparing actual output against expected output, is the best habit to get into sooner rather than later.

Truly, don’t do yourself a long-term disservice!

As a final point on this topic, correctness (passing each test case) is only one of four axes I use to assess their work. I also look at scope (attempting to implement all aspects of a program), design (efficiency of logic/algorithm(s) in a program), and style (readability vis-a-vis variable names, indentation, etc.). If you only grade on getting the code right, then I could see not providing test cases for every test case. However, if there is a bigger picture to assess in terms of code style and design, then I see no problem in giving as much information up front in order to aid student learning.

Improve this answer
$\endgroup$
2
$\begingroup$

I would say that all test cases don't need to be given, as long as the tests are not the only grading component.

If a teacher gives clear instructions for what a program should and should not do without specific test cases, students have to be able to come up with their own test cases which is good experience. Programming for test cases is never a good idea, and if that is what students are focusing on, it can lead to shortcuts or lazy code.

If a program fails a test case, the input or actions that failed should be given. Then a student has to run their program themselves to find out why it is failing. If given the test case instead, it is usually more obvious what was being tested for and therefore easier to find the error (which is less debugging experience).

Improve this answer
$\endgroup$
2
$\begingroup$

I think you should release test cases which test all of the functionality you want, but have private tests which test different different but similar inputs.

This ensures students have not just poked their code until exactly the tests have passed and then stopped. However, I find that when students are not given a test for functionality in advance they will often fail because they interpreted the spec differently -- it is in my experience extremely hard to pin down every corner in a specification, while saying to students "if in doubt, pass the tests" often helps clear up many corner cases.

Improve this answer
$\endgroup$
1
$\begingroup$

Some of it depends on the level of the student. There's obviously a big difference between a freshman just starting out, a senior about to graduate, and someone who's been in industry for awhile.

Keep in mind that, once your students are in industry, they're going to be expected to be able to write code that passes QA, and it's usually not the case that QA will supply them with a bunch of unit tests in advance. This being the case, it's essential that people learn to test their own stuff before they graduate.

I'm of the opinion that students should be taught how to write proper unit tests as early as possible. This should eventually progress to other kinds of automated testing over time as appropriate, such as Selenium or Coded UI tests. It's also useful to be aware of major testing techniques and terminology, such as boundary value analysis.

With this said, it's probably helpful to give new students tests early on and then slowly "wean" them off of that.

When I was in college, some of my instructors would also suggest test cases as part of the assignment.

I also agree with @EllenSpertus's suggestions for having students submit test cases as part of the assignment.

For learning purposes, you could start students out with "here's a piece of code, suggest 5 test cases for this" or "here's some code, write some unit tests for it." Again, you could start with giving them specific things to test for and progressively decreasing the amount you give them "up front" with the goal of eventually getting to "give me whatever automated tests you think are reasonable with your assignment" and grading them on how good and thorough the tests are.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.