How do you assess students' understanding of abstraction?

Question

We've just uploaded a video of a presentation Simon Peyton Jones and I did about Project Quantum to CAS TV. Project Quantum is an attempt to crowd-source low-stakes formative assessment items for computing; we've just over 5,500 questions at the moment.

I boldly (and perhaps erroneously) claim that there's no bit of the computing curriculum for which we can't create multiple choice questions (MCQs). Note, this is rather different from claiming we only need to use MCQs to assess learning in computing, which is very far from what I believe.

My colleague Pete Kemp remarks on the Facebook that he's "still waiting on a Q for abstraction that isn't about definitions...". Here's what we have on the Diagnostic Questions site at present (free registration required).

So how do you assess whether your pupils understand, or indeed can apply, abstraction? Could you knit this into an MCQ?

Examples (of far from satisfactory questions) from https://diagnosticquestions.com/Questions?CurrentSubjectId=1688&OrderBy=Newest&IsByStudent=False All CC-SA

Answer 1

tl;dr: Just say no.

This question is difficult on many levels. I seems to me to be a land mine of misconceptions and has the possibility to lead to poor teaching practice.

First the difficulties

1. Only concepts can be abstract. Abstraction is about ideas. Animal is a concept. Mammal is a concept. Animal is more abstract than Mammal since it contains the idea of Mammal, but isn't restricted to it. The kitty sitting on my hearth is concrete, not less abstract than Mammal. The kitty isn't a concept it is a thing.

2. Given some ot the examples, there is a misconception about what is abstract and what is concrete. Physical things are not abstract. One can't be more abstract than another. Even things that aren't physical, but which can be manipulated, are concrete, not abstract. A LinkedList (Java) isn't abstract. It is concrete. List, on the other hand (an interface) is abstract.

3. In computing there are many, many different kinds of abstraction. Even the concept of a while loop is an abstraction. In OOP languages classes implement abstractions. Interfaces (even if the language doesn't have that explicit feature) are abstractions. So, Python has interfaces, though it has no language feature with that name. If two classes have the same interface they are realizations of the abstraction, but neither is more or less abstract than the other. If one interface contains another interface then it is more abstract.But we also have data abstraction. Lists and Maps define an interface (an abstraction). They can be implemented/realized many ways. Each realization is concrete, not less abstract.

4. The web site cited seems to be trying to test everything with Multiple Choice Questions (MCQ). I question both the possibility of that and the wisdom of it. Some thing are complex MCQ are simple. Trying to push the assessment of complex things through such a small funnel seems misplaced to me - even foolish. I'm pretty sure that MCQ also advantage certain students and disadvantage others. They are fine for rote memory and facts, but pretty poor for deep understanding. I have nothing against their use unless it is overdone. MCQ are also difficult to validate. The wisdom of groups isn't enough. It takes statistical measurement and cross validation to do a good job of it.

5. The Font Construction example given in the question also seem to be testing specific vocabulary. This example threw me completely. I think I could eliminate one answer, but also think I could justify each of the others if given the chance. If it is specific vocabulary it is just rote memory and hoping that the student "remembered" to remember the right thing.

6. Often, getting a MCQ wrong and then later learning the correct answer can sometimes leave students simply puzzled and frustrated. This results in a destructive game that I'd rather not play.

7. MCQ try to put a lot of thinking into the questions so that no one needs to put any thinking into the answers after the student finishes. This may scale well, but is a sub-optimal form of education. After all, it is the students thoughts that we want to get a handle on in assessment.

I don't believe that a single, or even a small number of MCQ could capture the student's understanding of abstraction or any similarly deep concept. I think that if you had quite a lot of questions (say 20-50 of them) you might be able to get a handle on it, but it would take very careful cross evaluation of the answers given to be able to say with any confidence. Getting a single question wrong likely tells you almost nothing. And that, of course, makes MCQ less valuable for its main use - quick and easy evaluation.

I could, on the other hand create a bunch of questions that start to get at the student's understanding. The ones I'd like to use are short response questions, but some can be simple choices.

Assuming that the student have a common language background, something like "Is java.util.List abstract or concrete?" "Is java.util.Collection more or less abstract than java.util.Map". One choice for the answers needs to be neither of course.

You can show code fragments and ask whether they implement an abstraction properly or not. But a follow up "why or why not" gives you better information. And some of those questions would necessarily be a bit long to be meaningful and thus take some time for the student to evaluate.

"How can functions be considered abstractions?" Well, that was poorly stated, since a single function is concrete, not abstract, so you need to be careful: "How can the idea of a function be considered an abstraction?" is better. You could probably give a few suggestions for answers to this to turn it in to a MCQ, but it would be hard to come up with four, of which three were viable and yet not correct.

However, I've only hit the surface here and only explored a very small part of the abstraction space in computing.

Caveat: I never had to teach or evaluate a hundred or more students at a go. My classes were always small enough that I could use better evaluation techniques than MCQ, though would be likely to use a few in a larger assessment that allowed the students more creativity. If you do have to evaluate 800 students I hope you have help. If you don't something is deeply wrong. I note that Harvard CS 50 is run by a team of about 80 people, with quite a number of roles. I hope that each student has some individual person responsible for them. It isn't the professor.

My initial "just say no" is to the idea that you can completely evaluate a student with MCQ, not that they can't be useful, dangerous drug though they are. And note that I have the highest respect for Simon Peyton-Jones.

Answer 2

I would probably ask students to demonstrate their understanding of abstraction by writing code demonstrating solutions to a small problem at, say, three different levels of abstraction. I'd probably leave it to them to choose the levels of abstraction, and optionally write a short explanation of the differences in level of abstraction the code was intended to demonstrate.

I think multiple choice questions are highly effective for measuring awareness of facts. I see them as much less effective for determining understanding of ideas.

Looking at the specific questions you've posted, I think both of them are relatively poor. For example, when looking at a tennis racket, I suppose your intent was probably that "rectangle and circle" would be the correct answer. However, that's looking only at the outline of the racket. To a tennis player, the two parts that matter are the strings (which form squares) and the ball that it hits (a circle). So, rectangle and circle might be a good choice if you're thinking primarily in terms of computer graphics, but square and circle might be a better choice if you think in terms of a tennis racket's actual purpose.

Shapes, aren't really much of an abstraction. Shapes deal purely with the physical embodiment of the device. To test for abstract thinking, you need to start by thinking abstractly yourself.

If I were going to try to do this with multiple-choice questions, I'd probably have two in a row that contrast concrete and abstract thinking:

On a physical level (the shape of the concrete implementation), which of these most closely resembles a tennis racket?

1. Golf club
2. lollipop
3. automobile
4. tennis ball

On an abstract level, which of these most closely resembles a tennis racket?

1. Golf club
2. lollipop
3. automobile
4. tennis ball

In the first case, the correct answer is '2'. A tennis racket is shaped like a large lollipop.

On a more abstract level, a tennis racket is a tool for hitting a ball. So, at the more abstract level, the most similar is a golf club--also a tool for hitting a ball.

Answer 3

Fundamentally, I question the notion that you can test to see if students understand the idea of abstraction beyond a superficial level, even if you don't restrict yourself to asking just multiple choice questions. It's a little hard for me to articulate, but I sort of feel the best way to learn + get feedback on whether you understand abstraction is to actually try tackling a large, novel-ish problem and discuss your design decisions with a more experienced dev (or, if self-teaching, learn the hard way why some decision was a bad idea later down the line).

Nevertheless, here are some ideas (with the caveat that I have never once designed and administered a multiple-choice question style thing which means it's entirely possible I have no idea what I'm talking about).

These are all more CS-y and math-y then your example questions, but I figured you might as well address the skills you're after in a directly applicable setting instead of beating around the bush.

1. Test to see if students understand what implementation detail is -- that is, do the recognize when something "breaks" an abstraction?

   For example, you could give students a description of a car, and ask them to identify from a list which element is implementation detail. (For example, "hitting the brakes causes a moving car to slow down or stop" is part of the interface; "a combustion engine powers the car" or "the car moves via wheels" would both be implementation detail).

   If you want something more directly related to code, give them a header comment for some method or class, and ask them to do the same.

2. Test to see if students understand how to find similarities and find a viable abstraction between multiple items. (After all, if you have only one of something, why would you bother abstracting it?)

   For example, you could give students a description of several different queue-like objects (a FIFO queue, a priority queue, etc) and ask them which interface description would work for all of them. (This probably isn't the best example, but hopefully you get the idea).

   I would also add an option that says "none of these are good abstractions" and make that the right answer some of the time. After all, if students are taught that they should always be abstracting, then you'll end up with a bunch of architecture astronauts, which seems suboptimal.

3. Draw inspiration from category theory.

   After all, there's a argument to be made that constructs from category theory and design patterns are more or less the same thing (except that category theory is applicable to things beyond code and design patterns are more handwavy).

   For example, you could perhaps describe something like monoid (e.g. if the set M is a monoid, then it must be the case that that the set is closed under some associative binary operation and there must exist some identity element I in the set such that A op I = A and I op A = A).

   You could then explain how strings + string concatenation form a monoid as a case example (the empty string is the identity; concatenation is the associative binary operation), then ask them to correctly identify how lists or numbers or something are a monoid. You could also ask them which entry from a list wouldn't count as a monoid.

   This basically lets you test if students can understand how to use some (relatively basic) abstraction/adapt existing code to work with unfamiliar abstractions. (And tests to see if students know how to adapt to dealing with new definitions and such).

4. Give students a snippet of code that breaks an abstraction, and ask them to identify the correct reason why the code is wrong (and probably add an option for "no, everything's fine" to make it a little harder.)

Answer 4

Your tennis racket question, is not good. As tells as that is is abstraction, then focuses on the form, when the most important aspect of a tennis racket is its function. Therefore the answer is non of the above (those are all implementation detail, and you asked for an abstraction). A good abstraction is hitty thing.

The 2nd question, is very long, and can be summarised as, “guess the word I am thinking of”. While all answers are plausible, non are obviously wrong. Unless I grep my notes and see that “Thinking Abstractly” is the only one of these phrases that the teacher has used.

This is my attempt at an abstraction question.

Based on what I learnt on page 24 of Hello, World issue 3.

Which of these is an abstraction of a tennis racket?

• A hand held device for hitting a ball with. [correct answer]
• A colander. [A simile: they both have holes in, but do pupils know what a colander is?]
• A handle connected to a frame, with string tied across it, to form a surface for the ball to bounce off of. [An attempt at a detailed description, that is much longer than the other answers.]
• A tennis ball. [ a peer object ]

Answer 5

The best way to find out is to give programming problems, or similarly complex design problems, where you can look at the details of the solutions they create. I give lots of "lab" assignments where the students have to write programs based on a problem statement from the textbook. Part of the answer is: "Are they using what we just covered in class?", but the vital part is: "Are they gaining knowledge over time and learning to assemble things with increasingly complex structure?"

Some students clearly learn general concepts like how to define classes and use them sensibly, while other students struggle with this, or even more basic concepts like how to use variables in one UI method that were set in another UI method. (Could be Console or Windows, doesn't matter.) Some students take the ideas of Relational Comparison and Boolean combiners and use them correctly, others struggle and try to create things by sticking together the code they painfully got to work in the previous assignment (which they are now trying to re-use for an opposite case).

In short, assessing whether students have learned how to abstract is bone-obvious. Teaching it is the hard part. We need to find ways to teach abstraction. My research has turned up very little except arguments over what the word means, or worse: whether to start teaching at high levels of abstraction, or instead teach how computers actually work, etc. (See other questions on this site.) Poor start on a very significant problem!