How can we measure proficiency in programming languages?

Question

Recently Python and Java have started to be the de facto standard for teaching CS1/2. This means that students arrive in courses like Embedded Systems, Operating Systems or High Performance Computing without any experience with the C language. Where I teach we have a special "C marathon week" that provides an introduction to basic C programming to students who already code in Python and Java. We currently have an exam to try and measure students' mastery of C, but our approach is very empirical. We have impressions about what what they have learned during the first week, but very little of it is actually measurable. Mind you, students are able to do all the coursework, but we do not understand the relationship between their success and our introductory activity nor are able to quantify how much knowledge they gained during the semester (but after the introductory week).

So, are there best practices regarding evaluating their knowledge of C (or any other programming language) separately from their basic programming knowledge? This might generalize to other programming languages/technologies as well, so answers might not need to focus specifically on C.

Ideally, I am looking for something like concept inventories, but that focuses on the mastery of C and not on introductory programming.

Answer 1

I think you are taking a good approach with your boot camp. I'd only suggest a warm up exercise/mini-project in which students get some practice, perhaps writing in groups (pair-programming, perhaps).

One useful sort of project is called a fixer-upper. There is a pedagogical pattern by that name, actually. Give the students a program that you have written. Make it fairly large (bigger than they could write themselves). Make it a good a C program as you can. Then break it in a few ways. Some of the introduced errors are syntactical, which the compiler can catch. Others are more subtle and require thinking. The exercise is to fix the program. It is a good "hot" introduction to a language. In another course than those you suggest, I might give them a program that solves a problem they already know how to solve in, say, Python. But here, it might be better to do something that makes them think about memory.

But, I wouldn't worry overly about mastery at this level. Any "predictor" will be imperfect. Students can grow into the language along with the OS concepts, for example. I learned a lot about machine level programming in just such an environment.

The student needn't be perfect in one thing before they move on to the next.

But you may need to provide additional feedback or mini-lessons along the way if they seem to be a bit off track.

Answer 2

One type of assessment might be to focus on the key differences between their previous knowledge and the new programming language.

One key difference of C from Python is that C natively deals with a finite linear addressable memory. So one might test a students knowledge of that linearity (arithmetic on pointers and the potential for memory corruption), and finiteness (give them an algorithm that would explode in memory usage without GC, and have them fix it for use with a finite memory and no built-in GC).

There are other key difference that might or might not be as important or more important for you purposes. Test as appropriate.