I've seen too many cases when a student has completed a programming exercise (without cheating) but can't say, for example, what type of values a variable contains during program runtime (in case of Python programs).

It looks like the ability to solve exercises should not be the only (or even main?) criteria for assessment.

Do you know any research papers (or books or posts) that tackle this issue and/or recommend other assessment techniques / exercise types?

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – thesecretmaster
    Aug 26, 2017 at 10:49
  • 1
    $\begingroup$ IMHO, if a student can prove that their program satisfies a specification, they understand the program. Proof of correctness requires knowledge of types of values, conditions on values, and much more. $\endgroup$
    – beroal
    Sep 10, 2017 at 11:12
  • $\begingroup$ Code with pen and paper. It might sound odd at first but solves a lot of issues. Small exercises can basically be solved by brute force. Type a few things - run - make changes - run again - fix a few conditions - run again. Use smaller exercises and let them code with pen and paper. Ignore small mistakes like a single missing semicolon or minor typos. Now you can basically see what's in the head of your students and what you need to teach/address. $\endgroup$
    – wischi
    Jan 29, 2021 at 18:13

8 Answers 8


If I understand your problem correctly, it's that students can create programs that behave correctly without understanding why they behave correctly. I assume that they do this by some combination of brute force trying things and SO search.

Using program correctness for grading has the desirable qualities of being objective and automation friendly.

So how could you keep these qualities while testing for actual understanding? While I agree that good code structure, variable naming, and possibly comments can offer proof of understanding, they are all highly subjective.

What I would do instead is construct exam exercises that are relatively small requirements changes to homework assignments that students have completed earlier, ideally at least 2 weeks earlier. Then give each student access to the code (s)he submitted for the homework as a starting point for completing the exam exercises. Tune the original homework size and the exam requirements changes so that a student who understands the code and has well-written code can complete the exam exercise quickly, but one that started from scratch would not likely be able to complete the work. This accomplishes 4 things:

  1. It solves your original problem of testing whether students understand the code they have written.
  2. It maintains the objectivity and automation friendliness of grading on program behavior.
  3. It rewards production of readable, maintainable code.
  4. It prepares students for the world of coding outside of a class environment, where the majority of time spent is not on isolated exercises, but extending an existing code base.

Disclaimer: I work in industry and have no research to support this.

  • 5
    $\begingroup$ I'm also an engineer and not an instructor, but I'd be concerned about using specific homework assignments as a basis for an exam. If you have a student who struggled with the homework assignment, then they're already at a large disadvantage for the exam. Exam questions should probably stand on their own and the students are either able to answer them or unable to answer them. Instead of using the student's homework, simply provide your own program code instead. $\endgroup$
    – Ellesedil
    Aug 25, 2017 at 21:29
  • 1
    $\begingroup$ @Ellesedil It's true that students could be punished twice for doing a bad job on homework. I would mitigate that by not making too much depend on any one piece of homework or test question. Also, you could communicate how you test ahead of time, and have office hours or class time set aside for improving homework even after it has been turned in, which would improve the code for the exam, but not for the original homework grade. Generally, I think the issue can be managed/mitigated. Using your own code would be ok sometimes, but doesn't incent students to create readable, maintainable code. $\endgroup$ Aug 28, 2017 at 14:40

I have seen my share of this 'program gets output' but the programmer has no clue how she/he got there. It's funny how that happens so many times.

This is what I have done to at least handle the issue.

Before assessments, I break down the evaluation to include the following.

  • Simply getting the output gets them the bare passing grade.
  • Those who can use meaningful names for their variables, function calls get extra.
  • Those that can back up their loops, method calls and variable declarations with expansive ( 2 to 3 lines at least) comments get extra points.
  • In addition to that, those that can write psuedocode above the program code (perhaps in a big block of comments) get some more.

Obviously, this makes me their least favourite faculty in terms of evaluation and gets me a lot of complaints but I stick with it.

Update : Please note that, during evaluation, I don't expect all of them to be done. Say, a student is already using meaningful names everywhere, I would not mark her/him down for not writing comments.

  • $\begingroup$ Comments are for suggesting improvements; this conversation has been moved to chat. $\endgroup$
    – thesecretmaster
    Aug 26, 2017 at 10:46

I am wondering how much of this is because they can not express in natural language (don't know terminology). How much is because of just fiddle until it works programming.

Learning to express in natural language

This is important, to allow them to communicate with a larger team, to allow them to look stuff up on an internet search, and to answer some of the exam questions.

Programming by fiddling / tinkering

This is an important technique to use some of the time. However it should then be followed by evaluation.

For example: Trying to find the angle of a triangle, when drawing using turtle graphics.

  • Made a first guess (common first guesses: 30, 60, 45, 90, 100 )
    • Try it.
    • Decide higher of lower (this is often decided wrong, but that is ok)
    • Guess a new value.
  • repeat until you find the answer.
  • record answer with result for square.

Table of angles:

| number of sides | angle |
|        3        |       |
|        4        |  90   |
|        5        |       |
|        6        |       |
|        7        |       |

You can discuss what the best strategy is [binary search]. One you have a completed table (4 or 5 entries), you can start to look for a pattern. It there a general formula for a polygon. Also getting pupils to pretend to be the turtle can help, or teacher is turtle. In ether case another pupil is the instructor, that tells turtle to move and turn. While doing this they gain a deeper understanding of how the code works.


I'm afraid my answer here will suggest that you completely revamp how you teach.

The sort of problems that result in issues like this, seem to me to be problems that treat the computer as a fancy calculator. Problems given to students are of the "math-y" type. Some require tricky thinking, of course, but they are unlike the sorts of problems that people in the real world write programs to solve.

My suggestion, is to, instead, use simulation as the basis of your teaching. There are many ways to do this, but one of the best and easiest to introduce is the Greenfoot system. It provides both an IDE (for Java) and a simulation framework.

There is also an organization for users, Greenroom, who contribute simulation frameworks that you can start with and modify. There are hundreds of such simulations, some with teaching materials, even videos. Note that the Greenroom is a membership site. You will need to join.

Here are a few examples. I have used some of them, but not all.

Create Flappy Bird Game

Fuel Depot Question from APCS-A

Greeps - A Programming Competition One of the originals.

Karel J Robot meets Greenfoot A robot simulation - from the book.

2d Platformer Similar to Mario.

There are hundreds more, both at the main Greenfoot site and the Greenroom. Only the Greenroom requires (free) membership. With these sorts of frameworks and the programming that it involves, the problem you discuss simply won't arise.

It has been noted that I often promote Greenfoot. I'm not affiliated with the site or its developers, nor have I ever been. I know some of the developers and I have produced materials for use with Greenfoot. I also occasionally submit bug reports and feature requests to the site.

But Greenfoot aligns well with my general teaching philosophy and makes Object-Oriented Programming easy to teach to novices in an interesting and engaging way.


It's time for a little "code review." Have a student present his code in front of class and talk about how he made it work. Hey, this happens in the professional world. There is no time like the present to begin learning this vital skill.

  • 2
    $\begingroup$ @drewbenn - even if that was the case, id adds no value to someone's education to be forced to present his work in front of audience. And no, despite many claims of the opposite, you don't really do that when working as a programmer. $\endgroup$
    – Davor
    Aug 25, 2017 at 20:54
  • 1
    $\begingroup$ @drewbenn - unfortunately cs academics drown out actual people from the industry here, so the answers you get are the same nonsense you get in the academic setting in the first place. "Democracy" has severe downsides. $\endgroup$
    – Davor
    Aug 26, 2017 at 20:54
  • $\begingroup$ @davor Learning to present your ideas is an important skill. This often brings interesting discussion of design for programs that is both productive and useful. NCSSM has a nice environment; kids do not have anything to fear going in front of their peers. They are used to this. $\endgroup$ Aug 27, 2017 at 1:04
  • $\begingroup$ @ncmathsadist - brushing your teeth is also an important skill, but it still doesn't belong in CS curriculum. $\endgroup$
    – Davor
    Aug 27, 2017 at 9:44
  • $\begingroup$ It is well documented that we learn through experience, and that we often learn the most through teaching. What is lacking in many graduates is the ability for a developer to use the correct language to describe their logic and issues and importantly expectations. Talking about code out-loud should be encouraged and practised. $\endgroup$ Feb 3 at 11:33

I also encounter this issue (just encountered it yesterday in a lab exam). This is how I differentiate between someone who has done his work and just not been able to explain it and someone who has crammed/cheated:

  1. Deleting some part of code and asking student to rewrite it
  2. Asking student to modify a snippet of code
  3. Explain this line

Usually Point 2 is enough to differentiate. Kindly ensure to let him/her sit calmly in order to not let nerves get hold of him/her - it works for me!

  • 1
    $\begingroup$ I'm using the second point as the base for grading first-year computer science students. In the setting I usually teach, students prepare their solutions as a homework, so it's all too easy for them to just find a solution on the web; so they don't get a grade for just providing a solution that works. Most of their grade comes from implementing small modifications to their code (e.g. "Please change the order from ascending to descending" for an assignment on sorting) and bonus points for code clarity (if they can explain why; typical question is: "why is this identifier named this way?"). $\endgroup$
    – liori
    Aug 25, 2017 at 14:13
  • 1
    $\begingroup$ I got a very clear example of #1 back when I was a lab assistant. This was in the era of floppy disks, on single drive machines backup meant a lot of swapping. The idiotic backup routine couldn't sense the wrong disk in the drive and someone not paying attention could easily copy the target directory over the source--deleting everything. I knew how to rebuild them and thus got a fair number of students crying for help. (Continued) $\endgroup$ Aug 25, 2017 at 21:06
  • 1
    $\begingroup$ Everyone would freak when I sat them down in front of a binary dump of their data and asked them to identify it but after a bit of paging through it they would start recognizing pieces and I would build up the map I needed. One gal absolutely couldn't get it. Somehow it wasn't a surprise when later I found out she had copied everything. $\endgroup$ Aug 25, 2017 at 21:12

Isn't this what lab reports are for. Computer Science is a science and should come with some basic rigors. Copying some code and hacking away might get an output, but forcing students to do analysis, flow charts, etc will prove how much they really understand the process. Even simple things like writing a paragraph on the theory and another on the basic algorithm to solve the problem would go a long way.


Minimal: 'Provide a code solution', (does not necessarily need to be working).

Slightly better: 'Show how your answer works, or does not work, with detailed explanation and/or comments.

Best: 'Show how you tested your solution, describe what problems you found, and how you tried to debug and fix them'.

Awarding a large marks boost for details of testing and debugging renders SO/web copypasta answers much less useful. Testing and debugging are absolutely essential skills when developing software, and the sooner that students practice those skills, the better for us all.

The number of SO posts with non-trivial code, but demonstrating no debug skills whatsoever, is embarrassing. Too many cannot, (or will not), summon up enough programming skill to print out variables that may help diagnose a problem, never mind acually use a debugger. Educators should be pressing home the message 'If you cannot debug, you cannot develop programs'.


Not the answer you're looking for? Browse other questions tagged or ask your own question.