# Do you see the “Bimodal Distribution” too?

When people learn programming in a school setting, tests show a "two hump" tendency. Some students race ahead and grasp it relatively well and others struggle. Apparently this is more pronounced in programming than in other areas, like math or language. It seems to occur at all age levels and circumstances, and throughout a curriculum.

Do you observe this? If so, do you have any explanations, and any ways to address it? It seems to be particular to programming, so it is our problem to deal with.

• Is there data and/or research to support this? I don't necessarily disagree (and have some theories of my own), but I'd first like to read more about the fundamental premise to the question. – Peter Jun 17 '17 at 17:01
• This question could really benefit from more details or examples of this, in things you saw. Could you add some examples? – ItamarG3 Jun 17 '17 at 17:42
• Anecdotally, here is Berkeley's CS1 final grade distribution, which displays some level of bimodality. – George V. Williams Jun 17 '17 at 22:17
• @Peter I suppose I was not so much alluding to research, or actual test results, but instead to the perception that some students grasp the ideas at the beginning much faster and more completely. Others seem to struggle no matter how long they go on. It is not to do with 'intelligence' but more of a predisposition. – user737 Jun 18 '17 at 13:12
• Linear Algebra was the "reverse binomial" class for me, in college. I failed the class hard and for a while I actually tried to find people who were currently, or had previous, taken the course and asking them how they thought they were doing / did. I ran into one person who had gotten a C. Mind, my sample size was pretty small (about thirty people), but they all claimed observation of the reverse binomial as well. – Draco18s no longer trusts SE Jun 23 '17 at 17:06

Ah, the famous bimodal distribution in computer science!

When I took my first CS class in college, I frequently helped out a fellow student in my section who struggled mightily, spending unreasonably long amounts of time on seemingly simple labs. We made very little headway together. In spite of a semester-long effort bordering on the heroic, the student just couldn't seem to get programming.

I asked my professor about it late in the semester, and he said that there were a certain number of these students every semester, and that he didn't really know how to help them. He said that, if they didn't withdraw and kept working, he would let them go with grades of C instead of the Fs they actually earned on their exams.

I saw it again years later, as I started teaching my first computer science courses. In every class, there were some number of kids who just didn't get it.

And I was not alone! Others were seeing it, too, and there was even an unpublished research paper that started to make the rounds on the internet called The Camel has Two Humps, which asserted both the bimodal nature of the field and that they had created a test that could predict who would be able to "get" it. That paper was hardly the beginning of the story. In fact, professors have been investigating this bimodality for years.

Lately, however, the story has been changing. The author of the original unpublished study issued a partial retraction, and researchers began looking at far larger datasets of introductory CS grades. It turns out that that bimodal distrubution is not really bimodal after all.

# So what is going on?

When we teach programming, we want our kids to be able to program. So we test them on programming, and we want to heavily weight our grades on, well, whether or not they can obtain the proper output.

To illustrate this, let's take a hypothetical problem prompt, and some sample answers. The question, which will be in an introductory Java class, will read, "Create a function called makeN that, given a number n, fills and returns an array with $n$ sequential values".

public int makeN(){
array[n] = 0;
array++; // until its done
makeN = array;
return makeN;
}


public int[] makeN(int n){
int[] array;
for(n < array.length; n++)
array[n] = n;
}


public int[] makeN(int n){
int[] array = new array(n);
for(int x = 0; x < n.length; x++){
array[x] = x;
}
return array;
}


public int[] makeN(int n){
int[] array = new int[n];
for(int x = 0; x < array.length; x++){
array[x] = x;
}
}


There's a clear progression among these answers, and they represent vastly different levels of understanding. Unfortunately, our topic requires students to be able to pull together many different concepts in order to do almost anything. If we were to grade by whether the answer fulfills the prompt, they would all receive the same grade.

I actually have come to believe that, in a good rubric for a coding example, "actually produces the proper final output" should either not be present at all, or should be worth a miniscule amount of credit. A good rubric will capture attempts at portions of the algorithm, and successfully achieving steps of the algorithm.

This example above may have been extreme, and I doubt if most teachers would award exactly the same grade to answer 4 as they would to answer 1, I this cognitive bias certainly exists (I possess it myself), and it seems perfectly plausible to me that examples 1 and 2 might receive almost identical grades.

The trick to get out of bimodal thinking, then, is to break down the skills into smaller subskills, and break those down into further and further tiny understandings. This has two benefits: first, you see the clearer progression among your students, and two, head-scratchers like answer #1 become actionable, and you can see how to help the student to improve.

Student #1 may never quite get to a complete picture during your class - he or she clearly has a lot to learn! - but it is unreasonable to the student (and to you) to demand basically 100% comprehension in order to avoid failure. And taking a partial understanding as a partial success shouldn't just apply to the student. It's the teacher's success, too.

• In other words, if a code path is not tested then it's broken. I now start to grasp why introducing graduates to do validation (rather than demonstrating correct behaviour is possible) is such a challenge. – Sean Houlihane Jun 18 '17 at 10:16
• to be able to pull together many different concepts in order to do almost anything For sure. One of the classes I had in college featured a group project, I only vaguely remember the details, involved CGI web...stuff. Anyway, down to the wire my group couldn't get it to run, said "duck it" and submitted what we had anyway. We were only one of two groups to submit anything and received a C. – Draco18s no longer trusts SE Jun 23 '17 at 17:12

No, bimodality is a myth.

### Evidence That Computer Science Grades Are Not Bimodal

It is commonly thought that CS grades are bimodal. We statistically analyzed 778 distributions of final course grades from a large research university, and found only 5.8% of the distributions passed tests of multimodality. We then devised a psychology experiment to understand why CS educators believe their grades to be bimodal.

On the other hand, 100-level course are more likely to bimodal:

Of the 45 classes which were multimodal, 16 were 100- level classes (35%), 5 were 200-level (11%), 12 were 300-level (27%), and 12 were 400-level (27%). For comparison, in the full set of 778 classes, 171 were 100-level (22%), 165 were 200-level (21%), 243 were 300-level (31%), and 199 were 400-level (26%).

We can then break down the bimodality by course-level:

+--------------+-----------------+
| Course Level | Percent Bimodal |
+--------------+-----------------+
|          100 | 9%              |
|          200 | 3%              |
|          300 | 5%              |
|          400 | 6%              |
+--------------+-----------------+


Still, we see that even 100-level courses are unlikely to be bimodal.

• I would like to read about that "psychology experiment". My point was not so much about the actual grades (it is difficult for me to assign scores as low as the students understanding shows in many cases) as the idea that some few students really have difficulty taking on the early concepts. So the finding that bimodality shows more at the 100 level is what I am talking about. – user737 Jun 18 '17 at 13:28
• I think we should be careful concluding "bimodality is a myth" just because of that one paper. For one thing, it's only at one university. For another, the data is spread over 17 years (how do you know all the bimodal ones don't come from the past couple years, as CS has become more inclusive?). Also, there are clear biases we all know about, e.g. mentioned in Ben I's answer ("let them go with grades of C"). Just because the final grades are not bimodal doesn't mean student preparation levels are not. – David White Jul 9 '17 at 5:56
• I've also got some concerns about the stats in that paper. For example, they say "We used Hartigan's Dip Test, because it was the only one available in GNU R at the time of analysis" - I have no idea if it's the right tool for this type of thing. They also say "we only performed Hartigan's Dip Test on distributions where the kurtosis was less than 3" - because WIKIPEDIA says you can only be bimodal if kurtosis < 3. Seems VERY sketchy to me (and the wikipedia page doesn't link to legitimate, published sources either). – David White Jul 9 '17 at 5:59
• Finally, just because one test fails to reject the null hypothesis that the distribution is unimodal does not mean you can conclude it's unimodal. It could be bimodal in a way that this one test doesn't detect. Generally, we don't "accept the null" - we only "fail to reject" it. Good statistics should try many ways to reject it, but this paper tries only one. Now, I'm not much of a statistician, so please take my criticisms with a grain of salt, but please also take that paper with a grain of salt! – David White Jul 9 '17 at 6:04
• @DavidWhite you are a better Statistician than I. – user737 Jul 10 '17 at 12:27

There is this research paper "Learning edge momentum: A new account of outcomes in CS1" by Anthony Robins that is much focused on that subject...

From the abstract:

Compared to other subjects the typical introductory programming (CS1) course has higher than usual rates of both failing and high grades, creating a characteristic bimodal grade distribution. In this paper I explore two possible explanations. The conventional explanation has been that learners naturally fall into populations of programmers and non-programmers. A review of decades of research, however, finds little or no evidence to support this account. I propose an alternative explanation, the learning edge momentum (LEM) effect. This hypothesis is introduced by way of a simulated model of grade distributions, then grounded in the psychological and educational literature. LEM operates such that success in acquiring one concept makes learning other closely linked concepts easier (whereas failure makes it harder). This interaction between the way that people learn and the tightly integrated nature of the concepts comprising a programming language creates an inherent structural bias in CS1 which drives students towards extreme outcomes.

• This fits with my 'hooks' idea in teaching: present important concepts so that students have an easily-recalled hook to hang more things on as they go along. I want to know why CS is more this way than other fields. Because it is entirely made-up, unlike subjects grounded in the natural world? Because one needs a correct and useful "notional machine" to write programs? Because it is built up of a very narrow set of concepts? Because it requires a particular personality tendency? Studies just go back and forth, asserting and denying the problem. – user737 Jul 7 '17 at 19:13
• Fom the paper conclusions: "The LEM hypothesis suggests that the CS1 distribution arises not because our students are different, but because our subject material is different. In pedagogical terms this account highlights the crucial significance of the early stages of learning, where progress in acquiring initial concepts begins to establish momentum towards successful or unsuccessful final outcomes." ... Well... I guess this could also be valid for other subjects, as you said ... :( – Nuno Gil Fonseca Jul 7 '17 at 19:24

I do see a bimodal distribution in my intro CS classes, especially on harder tests. I also see something very similar when I teach statistics, and (to a slightly lesser extent) calculus. I think it has much more to do with student preparation coming in, rather than innate ability. I also think it has to do with "test anxiety," since it's much more pronounced on tests rather than on labs.

Are you aware of stereotype threat? This is the observation that, if you put a minority group into a situation where there is a stereotype about how they will perform, then their performance will bias towards the stereotype. For example, if students get the idea that your tests are testing their innate abilities in the subject, then students from groups that are usually perceived as "bad at" math/cs/stats (e.g. women, non-Asian minorities) will do worse, even when you control for actual differences in skill between students (e.g. via comparison to non-test grades in the class). This has been demonstrated by study after study. And the sad thing is, there's almost nothing you can do about it. It's not like you only "trigger" the stereotype threat by saying something when you hand the test out. The threat is already there in student minds.

For my classes, one thing I've done is get rid of "test" and replace them by "quizzes" (which are still worth 30% of the course grade). Each quiz is lower stakes (three equal one test), so I hoped the threat would be less. It doesn't seem to have worked. I also try to convince students in data science that no one is innately good at this, because it's a brand new field. This seems to have helped much more. I did the same in calculus, pitching the course as "totally different than what you did in high school", because of it's computational and conceptual focus. This seems to have helped. So, I guess my suggestion is to try and shake students out of the mindset that there are "those who can and those who can't", to de-emphasize exams, and to make sure the intro class is designed to also accommodate students with weaker high school backgrounds. This may reduce the bimodal distribution.

In my experience, some semesters I have had very clear bimodal grade distributions in introductory programming courses. Other semesters have seen something close to even distributions. Overall, my grades have show tendacy towards bimodal in introductory courses.

Certainly, "The Camel Has Two Humps" and "Learning Edge Momentum" are an important part of this discussion and I'm glad to see that they've already been mentioned (we've got some good people on this site!). At the ACM SIGCSE 2016 conference, there was a lively panel discussion focused on "Should all of your students be able to pass your programming course?", or some similar title. Many participants felt strongly that believing in the bi-modal distribution and the often related belief that "some students can learn this, but some students just can't" has been a convenient way for CS teachers dodge responsibility. Others had very strong feelings in other directions.

In my experience, it seems that some students show up for a first programming course already having good problem-solving skills (you may even want to expand that to computational thinking skills). Other students don't have these skills. For the students that do have these skills already, they are simply learning language and syntax to connect to concepts that they already have. But for students who don't start off with strong problem-solving skills, they are trying to learn two very different and difficult things simultaneously.

My analogy is this: I could take a class on how to speaking Chinese (I currently know nothing about the Chinese language). Alternatively, I could take a class that teaches the rules and strategies of the sport of Cricket (I currently know very little about the sport of Cricket). For those programming students who do not have good problem-solving skills walking in to the first class, and no prior programming experience, they are faced with trying to understand Cricket, but explained primarily in Chinese.

As someone who occasionally teaches introductory programming, I like to think that I teach problem-solving in the course, but I'm afraid I often focus too much on the syntax and mechanics of the language, and just assume that students will pick up on the problem-solving as we go along. And while teaching, I'm often tempted to believe that all students understand what we are doing as soon as I see 3 or 4 students nod consent.