60
$\begingroup$

In some CS courses (especially undergraduate and high school courses) the tests and the final exam are written on paper without the use of a computer. But if the students are in a CS course, shouldn't they be allowed to use the computer? If some mistakes are difficult enough to not be noticeable while writing code on a computer, they are nearly impossible to be found while writing code on paper.

So, my question is, are there any advantages in doing written tests on paper, and is this the best approach?

$\endgroup$
18
  • 2
    $\begingroup$ @GypsySpellweaver, typos occur all the time if you write your code on paper $\endgroup$
    – Safirah
    Jun 13, 2017 at 20:44
  • 13
    $\begingroup$ Dijkstra never used a computer. $\endgroup$ Jun 14, 2017 at 0:52
  • 34
    $\begingroup$ "Computer Science is no more about computers than astronomy is about telescopes." [Dijkstra]. Paper tests are still relevant for CS courses. $\endgroup$
    – apnorton
    Jun 14, 2017 at 2:43
  • 9
    $\begingroup$ IMO it depends on what you're teaching. Traditional CS is as much about theory as anything, so the knowledge and understanding is what's important. If you're teaching coding and/or for real world usage. (e.g. a 3 month intensive iOS program) then much of what's important is one's ability to use the tools available to them, so a paper test would likely be kind of terrible. $\endgroup$
    – GetSwifty
    Jun 14, 2017 at 17:02
  • 3
    $\begingroup$ Why do most answers assume that the only alternative to paper is a fully featured IDE and not a minimalistic plaintext editor like notepad? The biggest advantage of using a text editor over paper is that it supports insertions. $\endgroup$ Jun 16, 2017 at 8:41

24 Answers 24

50
$\begingroup$

Edit: This relatively high voted answer seems to be confusing to some, as it does not seem to give a clear answer. TL;DR: yes, doing tests on paper is usual at least in my university, at least many years ago. My anecdotal experience means to give a background where this might be coming from, together with the title "CS!=coding". The last section at the end gives my opinion on why it is a good thing.

CS != coding

You also have to acknowledge that CS is not coding; coding is not CS. You can have great coders who have not the faintest idea about CS (and vice versa: rubdos mentions the example of Alan Turing himself, who died around the time the first programming languages were invented). I can't remember whether in my CS studies I ever wrote actual real world code in a test. Lambda calculus, sure, and other formal stuff (more like mathematic formulae/proofs than programming).

I do clearly remember that coding/programming was not part of the curriculum. It was assumed that people who went to study CS already were able to program. At that time, Java 1 was around the corner, and my university offered a voluntary 2-week Java crash course before the actual semester started. I did not take part (having a few other real world programming languages under my belt at that time), and never missed it. There was a bit of programming during some practical courses, but here also the profs assumed that everything we needed to know we'd pick up ourselves. And right they were.

Funnily enough, one of my profs was very active in the Java community, and used it to motivate proofs in temporal logic / correctness. But that was very much theoretical, about concepts and principles - "coding" had nothing to do with it whatsoever. It was more about "how do you create a compiler" instead of "how to you write a program". That's what university level CS is about, after all.

Hence, as it was not part of the curriculum (being assumed to be known as a matter of course), no need to ask about it in a test; hence no reason to use a computer in tests, not any more than in any other field.

EDIT: this answer is my experience from university level CS education; it somewhat applies to earlier levels as well, in my opinion - it is more important to teach the principles than the actual coding; and a test would be better served with checking that they understand roughly what it's all about instead of writing perfect syntactic code with a pencil. I'd rather have a pupil explain what a basic design pattern is, instead of showing the actual code for a simple for loop. Programming languages come and go, principles stay (adjust to your own curriculum of course).

$\endgroup$
15
  • 2
    $\begingroup$ I had a similar experience. We had some Pascal in the first year and after that professors assumed we are able to code: You have to do some exercises on operating systems? Go get a C book and learn it yourself. $\endgroup$
    – Giorgio
    Jun 14, 2017 at 17:22
  • 2
    $\begingroup$ so.... should they do their tests on paper or not? $\endgroup$
    – Viliami
    Jun 16, 2017 at 10:06
  • 2
    $\begingroup$ @BrianR "handwriting LISP is a massive pain" -- random fact: LISP is the only language currently in widespread use that was originally designed to be handwritten. $\endgroup$
    – Jules
    Jun 16, 2017 at 10:11
  • 2
    $\begingroup$ Most of my CompSci education was theoretical. There were assignments; write code, get it working, submit it, etc. That's not CompSci; that's the APPLICATION OF CompSci. How does a red / black tree work? How is searching it O(log (n))? If you can't explain it, how can you apply it? My Database class dealt, extensively, with relational model. Most of it was done using algebraic notation. Once I had that, SQL was a breeze (2 class periods). Today, I write pretty complex SQL. "An exceptionally good mastery of one's native tongue is the most vital asset of a competent programmer" -- Dijkstra $\endgroup$
    – Meower68
    Jun 18, 2017 at 15:48
  • 2
    $\begingroup$ @Skipp Computer Science is about computers in the same way Astronomy is about Telescopes (misquote of "Edsger W. Dijkstra"). Would you say "astronomy is a superset of telescope operation"? $\endgroup$
    – Yakk
    Jun 19, 2017 at 6:04
54
$\begingroup$

100% yes. In beginning courses, it's practically a cognitive requirement. Let me see if I can break down for you why it is so important. It comes down to what we can know that the student knows.

Let's say we have a student who is a rather mediocre coder and who, to be honest, really has no idea what is going on in class. We're going to give him a problem in Java, which is to go through an array of ints, add all of the numbers in said array, and return the result. We're also going to let him use an IDE to build it, so that he can find syntax errors and spot-check himself for silly errors by running the code. The first thing he writes is this:

for(z=array[0], z < array, z+1){
    z=array[0];
}

So, how much can we say that he actually knows? I can forgive that he used commas in his for loop. I might even, depending on his background, forgive that he forgot to declare z. But what can be said about the fact that he set z to array[0] instead of to 0? Or that he is comparing z to array instead of array.length(), or that he is never actually changing z in his z+1 statement? These are mistakes of comprehension, not syntax. What, also, can I say about the fact that he keeps setting z to array[0] over and over again?

The fact is that, if he writes this into an IDE, a few things will happen. A bunch of code will light up red. A red line will remind him that he forgot to make a return. Another red line will tell him that z < array is not possible, because they are of different types. So, he will start typing things until the red lines go away. Does he necessarily understand the things he is changing, or has he been trained, Pavlovian-like, to change code that has been underlined in red?

As he makes some red lines disappear, new red lines pop up. He then turns his attention to those until he makes all the red disappear, and now he has this:

int g = 0;
for(int z=array[0]; z<array.length; z=z+1){
    g=array[0];
}
return g;

This code is much, much better. It's still not perfect. But, given the flaws in his first draft, do we have any reason to believe that he really understands this new version and why it is actually better? Or is he just changing code until the red goes away?

Now, he runs it, and it immediately ends by returning the value 0. So, since he still has about 5 minutes left to his exam time, he starts just changing things at random. He changes his int z to be = 0. Now he starts getting a number returned! He tries putting g into g=array[0], then he puts z into it. Now he has g=array[z].

He dithers about a bit, and finally adds a + in there. He now has the correct code:

int g = 0;
for(int z=0; z<array.length; z=z+1){
    g+=array[z];
}
return g;

He turns it in to the teacher, who says, "Boy, he sure understood this! That's one student that I don't need to give any extra attention to, because he is in great shape in my class!"

The takeaway questions then are:

  1. How well does the student understand the material?
  2. How well does the student's grade reflect that?
  3. How much worse are the odds that the student will now receive the extra help that he so desperately needs?

We need tests that actually show what you really know, which is not always the same thing as what you can produce with prompting help from an IDE.

$\endgroup$
18
  • 7
    $\begingroup$ +1 Great motivational answer. It would almost be funny if that were not what is actually happening in the real world (in businesses) a lot, in practice - programming by trial&error. $\endgroup$
    – AnoE
    Jun 14, 2017 at 13:01
  • 16
    $\begingroup$ @BenI.: As a maintenance coder of 17 years' experience digging through other people's code, I can say, this is exactly what is going on far too often. $\endgroup$
    – DevSolar
    Jun 14, 2017 at 13:16
  • 7
    $\begingroup$ As sad as it is, that's very common. Even worse, the offenders are usually "schooled" while the ones not doing this are typically self-educated. The first group learn to pass classes, the latter learned the language and how to use it. $\endgroup$ Jun 14, 2017 at 16:35
  • 12
    $\begingroup$ I learned to program long ago without an IDE. (Shoot, I did some of my learning on punched cards and teletype.) When my co-instructor pointed out that our beginning students seem to just chase the wiggly red lines around the screen, I was horrified. Visual Studio should have a mode that lets you just get something typed in before it starts trying to 'help'. I have seen students take nearly perfect code and turn it in to garbage because they were trying to solve the wiggly lines instead of coding their solution! This is completely aggravating. How did we end up in this place? $\endgroup$
    – user737
    Jun 14, 2017 at 20:17
  • 7
    $\begingroup$ "Does he necessarily understand the things he is changing, or has he been trained, Pavlovian-like, to change code that has been underlined in red" -- this is an eye-opener. +1 $\endgroup$
    – Firelord
    Jun 15, 2017 at 8:10
24
$\begingroup$

Although most of the answers here seem to be in agreement that coding tests should be done on paper, I would like to offer a different opinion. Any test that includes coding would benefit the student, and most likely the teacher, if it was done on a computer with an IDE for the appropriate language rather than on paper.

Since the question contains the tags for High School and Undergrad, I will focus on tests which are common in lower level programming classes which are often focused on a single language and contain significant amounts of coding based questions.

Here are a few of reasons:

  • Writing is messy - Not everyone has the best penmanship skills. This is not just an issue of readability, on a paper test space is often a premium and so creates an issue for anyone who is not able to write small enough to fit the entire solution in the space provided. I have had at least a few instances I can recall where I miscalculated the number of rows my solution would need, prompting me to either erase parts of my code to make the writing more condensed, or else doing an awkward scrawl in the corner to fit the last line. This could be solved by providing sufficient space for the question, but since this is completely reliant on the instructor's assumptions about the intended solution and the size of the writing this assumption rarely hits the mark.
  • Writing is slow - It's been a while since I sat down and wrote anything of any significant length, but it's always painfully apparent whenever I do that my typing speed has far outpaced my ability to write. Not to mention that after a few pages, my hand starts to cramp from the unusual strain. In the age of smart phones and PCs, I imagine this is representative of the majority of the population, and should be moreso for CS focused individuals. Tests are timed. Forcing a student to use a slower method of recording information adds additional strain which can resolve itself in many different ways. Typos, bad naming practices, less readable code, take your pick. Worse, it creates a sense of pressure, where even if you know exactly what you need to write for a perfect answer you are limited by how fast you can get it down on the paper.
  • Writing is not representative of their past experience in the course - I will lean on my undergrad teaching assistant experience here to say that students rarely have to write any code on paper throughout the course, except for tests. All homework assignments are submitted online to ensure the submission timestamp matches the due date. Instructor notes that include code are available online to streamline the lecture process. All coding is done on the computers in the lab portion of the course. Major coding assignments are also done on the computer, as the first check(and a decent chunk of the grade) is "Does it run?". And then they get to the test and suddenly all those skills and muscle memory they've built up over the course of the semester is not directly applicable to the problem at hand. There are too many things that IDEs simplify that simply go unnoticed until you have to do without. And while some may say that that is exactly how it should be during the learning process...
  • Writing is not representative of real world experience - I can happily say that besides a couple of interviews, I have not had to write out my code out on a sheet of paper. I have used a white board on occasion to brainstorm ideas or write out chunks of code for others, but I was not required to present a solution that was polished enough to work "as is" if typed into a computer. Because it's not the syntax that's at the crux of the job. It's the ability to reason through and come up with a solution to a particular problem within the bounds of the programming language(s) you are using. The syntax is secondary. It's why APIs, and other manuals exist out there. It is not your job to memorize the names of the methods of all the libraries that are available to you. All you need is to know which does what, and how to best leverage them to achieve the solution you are trying to implement. Unfortunately, written tests treat syntax as part of the grade. And if you can't recall the name of that one conversion method part of the Integer class? That will usually cost you quite the chunk of points.

So, in conclusion, I believe there are significant downsides to forcing students to take tests that require coding solutions in the written format. Paper tests enforce memorization and a focus on the syntax of the language rather than concentrating on the problem solving and abstract thinking that is at the heart of programming.

This answer is based on my experience with the US educational system, and so may not be directly applicable to other countries.

$\endgroup$
14
  • 4
    $\begingroup$ Upvoted because I believe this contributes to the conversation, though I obviously disagree with your conclusion. How do you get around the problems in my post and in @AnoE? $\endgroup$
    – Ben I.
    Jun 14, 2017 at 16:49
  • 4
    $\begingroup$ I think some of the criticisms you have (it's easy to get syntax wrong/not remembering certain methods can be fatal) can be solved by simply being more lenient with minor syntax issues while grading and by providing students with a cheatsheet listing all the built-in methods they might need to use (and perhaps even providing examples of syntax). $\endgroup$ Jun 14, 2017 at 17:53
  • 3
    $\begingroup$ Another potential problem with doing tests on computers is it makes it easier to cheat, unless the instructor has full control over the computers being used, which is often infeasible due to budget constraints or large class sizes. If the school has a competitive majors system and the intro course is an important part of admissions, that only compounds the problem. (And of course, this is always a potential problem in high school, since students want high grades for college admissions). $\endgroup$ Jun 14, 2017 at 17:55
  • 5
    $\begingroup$ I'd also like to point out that especially in US colleges/high schools, students are expected to take a variety of other classes in other disciplines unrelated to their primary interest as a part of a 'general education' requirement (literature, history, other liberal arts classes). It's common in such classes to require students to write essays and short answers on papers during exams. As a result, I think it's reasonable that STEM classes can expect students to do a comparable amount of writing on their exams. $\endgroup$ Jun 14, 2017 at 18:01
  • 5
    $\begingroup$ I write on the whiteboard at the front of class, and my writing is poor. Real programmers write on whiteboards in meetings. If they cannot sketch out a solution, they cannot be part of a team. $\endgroup$
    – user737
    Jun 14, 2017 at 20:20
19
$\begingroup$

Speaking as a former student and as a coder: yes.

Perhaps there's an element of in-my-day geezerism here, but under the assumption that by "on a computer" you mean "in an IDE" I'll point out that when you have to write it out, you don't get:

  • syntax highlighting
  • autocomplete
  • as-you-type compiler checking
  • new project wizards ("What was the signature for main again?")
  • ...

Remembering to end every line with a semicolon without the benefit of syntax highlighting may seem like tedious detail, but if by the final you haven't picked up that habit, doesn't that indicate something about you as a coder?

If you say that as a teacher you don't care about the semicolons, you just want to see the algorithm, then fine. Don't count off when they're missing. But don't let it be a "gimme".

$\endgroup$
8
  • 1
    $\begingroup$ But do you think that this also applies to more advanced programming courses? $\endgroup$
    – Safirah
    Jun 13, 2017 at 20:53
  • 1
    $\begingroup$ @Safirah What's your definition of 'advanced programming course'? $\endgroup$
    – Pikalek
    Jun 13, 2017 at 21:05
  • 2
    $\begingroup$ Will you say the same if the students are given some normal text editors with ability to compile rather than IDEs ? $\endgroup$
    – user656
    Jun 14, 2017 at 0:30
  • 1
    $\begingroup$ As far as I'm concerned programmers shouldn't worry about things the system is going to deal with. I don't memorize anything like all the signatures--I know basically what's going to be wanted but let the IDE show me the details for anything I don't use all the time. $\endgroup$ Jun 14, 2017 at 4:24
  • 5
    $\begingroup$ As a uni student and programmer, in my experience the courses have two parts. The first is assignments and/or computerised tests. These are throughout the course, heavy in programming and done on a computer, usually at home or in labs. The second is exams. These are the end of semester ones. They almost never expect you to write code and rarely expect you to read code. This is because the course tests your coding ability through other parts of the course. The exam is about verifying that you understand the concepts that they have been teaching in the course, so are typically wordy answers. $\endgroup$ Jun 14, 2017 at 7:51
14
$\begingroup$

As a student who went through 4 years of undergraduate in computer science, writing all my exams on paper, I do think there are advantages to a paper exam.

  • If you need an IDE to tell you the things that are missing, then do you really know your material and syntax?.

  • Having an environment to test your code over and over isn't really practical for examination purposes or the real world. You should understand the obvious cases and edge cases while writing your solution.

  • As a lot has already mentioned, CS is not purely coding. From personal experience, I would say my undergrad was 75/25 split between theory/coding. Anyone can code, but learning the theory behind algorithms and best practices is why we are computer scientists.

  • Continuing on the point above, courses with heavy programming involved usually had a 60/40 split on grading where 60% weight is from code and only 40% from paper exam.

  • It is a lot easier to ask high level questions such as draw a 2-3-4 Tree, how a certain step in heapify or graph theory looks on paper than on a computer.

To address your point of mistakes being too difficult to detect even with a computer, then it is even harder on paper. No, as a graduate and TA, I can tell you when I'm grading first/second year exams, I can clearly see if they do not understand a certain subject or leaving out edge cases. We do not deduct marks for brace brackets, semi-colons, etc.

In conclusion, I think you are mixing the concept of coding challenges such as ones from Leet Code where it is purely coding based, with a discipline that incorporate a lot of theory and algorithms

$\endgroup$
2
  • 1
    $\begingroup$ I appreciate getting a student perspective. Thank you for posting. $\endgroup$ Jun 14, 2017 at 21:18
  • 1
    $\begingroup$ I agree, there is a definite difference between concepts and code. Concepts should definitely be done on paper. It is far easier to sketch a diagram etc than it is to answer on a computer. For programming tests though, doing it on paper is awful. $\endgroup$
    – Baldrickk
    Jun 21, 2017 at 11:13
12
$\begingroup$

Speaking as a former student, a coder & a teacher: yes.

In addition to the points raised by G. Ann, I would add the following:

  • Problems on tests are typically far simpler than full real world problems. It's reasonable to expect increased attention to detail (missing semicolons, etc) when there are fewer details to attend to.
  • In a writing class with long answer question, I would expect proper use of grammar & punctuation. Common programming syntax is not different enough to warrant special treatment.
  • The instructor is free to adjust the distribution of points as they see fit. I typically gave most of the points if the general shape of the solution was correct & held some points back for students who could provide compile ready answers. Different course / class /teacher combinations may have different needs. Good teachers adjust things as needed to provide the best possible learning experience they can provide.

Regarding OP's comment about advanced course - I found advanced courses don't lend themselves well to asking programming questions on exams. Students should have covered programming concepts in intro courses. I prioritize concepts over implementation details in upper course exams & leave the implementation details for implementation assignments / projects.

$\endgroup$
12
$\begingroup$

Speaking as a former student, a former competitive programmer, a real world programmer, yes.

Forcing student to write code on paper is not pointless. It has following benefits:

  • Giving you syntax awareness, avoid computer generated code. Therefore giving you deeper understanding of the concept.
  • You will find a way to write shorter code, since you can't copy-paste.
  • You will be more careful, since you can't undo/redo. You will develop the habbit of drawing the concept before going into the code. My personal experience is, don't open IDE before you have a solid solution in paper.
  • Pen & paper is always nice for sketching up idea. You should get used to them.
$\endgroup$
4
  • 4
    $\begingroup$ You have to acknowledge that for some (frequently extremely talented) individuals, this forcing will be stressful and counterproductive - but as a generalisation you make very valid points. $\endgroup$ Jun 14, 2017 at 9:18
  • $\begingroup$ The pen is mightier than the MSWord. (the pencil is mightier than the pen: it has an eraser.) $\endgroup$
    – user737
    Jun 14, 2017 at 20:18
  • $\begingroup$ @SeanHoulihane, I agree. $\endgroup$
    – hucancode
    Jun 15, 2017 at 8:38
  • $\begingroup$ @SeanHoulihane The number of folks who occupy that category are greatly outnumbered by those who think they occupy that category. $\endgroup$ Jun 15, 2017 at 19:54
12
$\begingroup$

Maybe some complementary points to what as already been said (as a CS assistant professor, giving both theoretical and practical courses).

  • Even for practical courses (those which involve coding), writing code on paper is good. The point is not to fill your page with boilerplate code (and I cannot stress that enough). Writing sequence of Java imports or mystical incantations to setup say a Map/Reduce transformation in Hadoop on paper makes absolutely zero sense. However, being able to write correct application logic and being aware of implementation aspects (some object can actually be null, problems with side-effects...) is a must. The hard part (which is actually an art I have not fully mastered yet), is to make exams that tests theses aspects without the noise and being of course lenient about small syntactic mistakes (missing ; capitalization and so on). At some level, what I want to test is this: "Here is a problem (from which I have abstracted the noise for you). Can you solve it with programming, without having to compile it 10 times until there are no more syntax/type errors and without running it a 100 times with printfs all around to see why this does not work ?". Because if the student can do that, then I know (s)he will be able to take it to the next step and generalize it to harder problems or use it in other contexts. If a student takes whatever is in the lecture notes, paste it in an IDE and fiddle with it until it somehow works, I cannot make the same guarantee. Also, you cannot copy/paste on paper, which forces you to think about your code and write it in a terser manner. I have to initialize an array. Ok I'll write a loop rather than cut/and paste 25 lines and change each one.

  • Conversely, even for theoretical classes (complexity/calculability, type-systems, logic , ...), I think it is good to have a bit of coding. Write a lambda calculus interpreter. A small (even naive) SMT solver. Code several sorting algorithms and try to see what constants are hidden inside your big O. Is the impact of sorting in place noticeable without resorting to having to copy your data? In highly advanced courses (in particular type systems and logic) you only really see what's going on when you can test large examples. Pen and paper only take you so far.

$\endgroup$
10
$\begingroup$

Your question has an implicit assumption: that CS is programming and what you're learning (and will always learn) in CS is programming. That's not necessarily true. Even though an introductory course "teaches" a language, probably something like Python, C/C++, or Java, in actuality you're learning a type of programming language, a procedural language (control flow) or OO. Ask someone who's learned Scheme as their first programming language how different it is from C.

Following on from intro, a data/discrete structures class (the second class in CS) could be taught language agnostically, but really you're learning things like sets, relations, and combinatorics which can all be tested about conceptually. Anything beyond discrete depends on the class. An OS or networking class could have C as the required language since it's "systems" class, along with an upper-level undergraduate/first-year graduate database class. And exams in those classes aren't about programming, rather more along the lines of "When is a semaphore better than a mutex?" or "What's CIDR and why is it important?" If your DB professor is unkind, they'll ask you to give pseudo-code for a B+ tree and ask you a relational algebra question. Graphics sometimes introduces OpenGL, but concepts are taught and tested mathematically, something along the line of given a 8x8 grid of values, what's the interpolated value at some point in the grid. Volume rendering is expressed as solving an integral. Image processing can be taught as a signal class, i.e. use PDEs. Scientific computing could ask why is Householder more stable than Gram-Schmidt.

Turning towards other CS things, algorithms (runtime complexity) and theory (abstract machines, automata) are all done with pencil and paper. Ultimately, the higher you go, the less programming is taught and more concepts are taught and then expressed through programming.

TL;DR Yes.

$\endgroup$
3
  • 3
    $\begingroup$ this answer does answer the question, but it's not entirely to the question. The question asks about written tests about code, and not about cs in general. You are right that CS is more than just code, but that is not relevant to the question. $\endgroup$
    – ItamarG3
    Jun 14, 2017 at 6:20
  • $\begingroup$ @ItamarGreen maybe the implication is: "if it is not a coding question, it should be answered on paper"-? $\endgroup$
    – user737
    Jun 14, 2017 at 20:25
  • 2
    $\begingroup$ +1 (+2 if possible) for the first paragraph. I suspect this is whymy first language at Uni was gcl, for which there was no compiler. $\endgroup$
    – Hennes
    Jun 16, 2017 at 16:48
10
$\begingroup$

Having taken and TA'd for classes which use both approaches, I would have to say that it all depends on what you want to see from the students.

On the side of having paper exams, it has already been mentioned that CS!=Programming. If you're trying to test concepts of CS, there doesn't have to be a single line of code written on the exam, and a paper exam will probably give you a lot more insight into how a student is doing on comprehension than allowing them to use an IDE during an exam.

In that vein, you can ask:

  • multiple-choice questions (hard to write well, but can be extremely useful in large classroom situations)
  • open-ended "explain to me how you would think about this" questions (some people mentioned handwriting being a problem, but in grading 100's of students' papers, I never found one I really had a hard time reading)
  • human-computer "what would the state of the program be after X iterations" questions (tests recall knowledge, which is sometimes all you need to tell how well they're picking up a specific language)
  • fill-in-the-blanks in code, e.g. "I'm trying to write a summation program; what needs to go in the blanks for the program to compile and run correctly?" (allows you to test specifics of syntax knowledge without worrying whether they're just re-enacting Ben's scenario)

On the side of computer exams, I've had a few of those too, and I think they can be useful in making sure your programming assignments actually taught what you meant to teach. I had a Data Structures professor who gave us so much sample code "to get us started" on our homework that most of the class failed and had to retake the final exam which was done on a computer, in an IDE. The class had gotten used to waiting for the "starter code" before starting their homework, so when the final exam asked us to write a stack instead of the linked-list we had written for the homework, most of them were stumped.

If you really want to have "programming" exams (possibly to test the sort of behavior I just mentioned), then I would suggest that comments should count for a large portion of the grade. This does two things: it allows students who are not quite as fast at churning out code to explain their solution, and it makes it so that that inevitable debugging nightmare that you know to look for but they don't doesn't have to completely tank their grade. It also gets them in the habit of commenting in code, so please, for the sake of every professional developer out there, grade their comments!

In my experience, it's a lot harder to write, and to grade, a "program this now" type of test that actually tests the concepts of CS than a paper one. As I said, I have one very salient exception to that rule in my experience, and if you're worried about academic dishonesty, programming tests on a computer that can't access the Internet are sadly about the only thing you can do. But if you have the option of a paper exam, it's usually a better fit for the subject.

$\endgroup$
2
  • $\begingroup$ Very good points made here. I have seen that method of providing "starter code" but I never took the bait. It just felt weird to me. $\endgroup$
    – user737
    Jun 14, 2017 at 23:47
  • 1
    $\begingroup$ Instead of "comments", I would consider outright asking for a (non necessarily super formal) proof of correctness. I value those immensely because I remember how I used to program before those were hammered in my head in Programming 101. It was not pretty. $\endgroup$ Jun 16, 2017 at 19:51
9
$\begingroup$

As much as I hate the fact that we need to consider this, paper and pencil makes it harder to cheat. Even if you restrict them to using lab computers and not their own laptops, and cut off networking to the room, it'd be easy to smuggle in a USB key with entire books worth of notes and the source code from every one of their labs and projects. (Yes, with enough effort, you could lock down the computers to the point where they can't even mount a USB key, but the logistics are a lot trickier to manage than just scanning the room periodically to see if anyone's got a small library of cheat sheets on their desk.)

(I don't think that this should be the only factor considered, or even a deciding one, but other answers have covered various pros and cons well enough already, and this is a factor that hasn't been mentioned yet.)

$\endgroup$
3
  • 1
    $\begingroup$ Good point. It is a factor, so far overlooked. $\endgroup$ Jun 16, 2017 at 2:19
  • $\begingroup$ I don't know - having the K&R on hand and the previous homework seems borderline acceptable to me, under the assumption that the test is not essentially a rehash of homework but offers instead an entirely new challenge. And, are you telling me disabling USB ports on lab machines isn't the standard already? :) $\endgroup$ Jun 17, 2017 at 17:34
  • $\begingroup$ @TobiaTesan If you decide it's acceptable, then you're naturally free to allow it. But so long as there's any amount of reference material that you don't want to allow, there's a benefit to making it harder to smuggle in. As for USB ports being disabled by default, that probably varies from place to place. We certainly don't do it here; we want students to be able to backup their work. $\endgroup$
    – Ray
    Jun 19, 2017 at 22:27
8
$\begingroup$

When I went to college, we had 2 exams. A practical and a theory exam. The practical was on the computer and we could use all resources including the internet. The theory exam was basically what you describe a paper and pen closed book exam where we would either write code or find the error in some pre-written. There were also standard question as well.

I see value in both methods. If I were teaching a course, I would probably do the same thing. Give a relatively hard practical exam (because students have all resources at their disposal) and also give them a written test. This will not only test their memory of syntax, but also give them a chance to really build something.

$\endgroup$
1
  • 1
    $\begingroup$ My students have a 'theory' exam on paper in my course tomorrow. Cross all 1010 of your fingers... $\endgroup$
    – user737
    Jun 14, 2017 at 20:22
8
$\begingroup$

I think this question is perhaps mis-framed or at least misleadingly framed. Fundamentally, the question an educator has to answer is: what knowledge do you want your students to demonstrate? Once you know that, the next question is how do you strip away all other distractions so that whatever product you evaluate gives you a good proxy for what knowledge your students actually acquired? The real issue here is how to tailor the evaluation medium to the desired learning outcome.

I think written-response tests (regardless of whether they are "written" with Word or a pencil) are tremendously valuable because they let you probe the degree to which students understand the theory. For the same reason, I think significant programming assignments are also tremendously valuable because they offer different information: is the student able to translate theory into working code? This approach gives you a much broader view of the student and gives you the opportunity to understand any inconsistencies. E.g: a student does awesome on coding assignments but sucks at written tests, maybe there's a learning disability that could be better accommodated?

$\endgroup$
6
$\begingroup$

If some mistakes are difficult enough to not be noticeable while writing
code on a computer, they are nearly impossible to be found while writing code on paper

I don't see how this follows in general.

It depends, of course, on what the test is testing: if the test is about finding the one incorrect value in a 2000-entry trig table I would expect a computer would be of help, but that's not what a CS test usually looks like, is it?

I expect a good test would in general measure the student's understanding of a topic, not their ability to mechanically carry out repetitive tasks.

So, my question is, are there any advantages in doing written tests on paper, and is this the best approach?

I am going to take a shot in the dark here and make an assumption about your question: your question is about programming and the "test" is writing a program.

Under that assumption I would answer with a loud "yes" - you'd want students to be able to reason about programs and convince themselves of the soundness of their reasoning, not just go by trial and error until they converge to a working program by sheer chance.

That's, after all, no way to program in real life.

On the other hand, I would also have a lab test to measure the student's ability to actually compile and run a program and/or launch a debugger - after all, once you've produced a correct program you still need to compile it to be useful...

$\endgroup$
2
  • 4
    $\begingroup$ One sign I notice when students don't quite have the knowledge to solve a problem is that they really double down on things that they DO know. They become very innovative and churn out code that plays around the edges of the problem, enhancing the I/O and things, when a 10 line program would have demonstrated the concept adequately. I guess there would be less temptation to do this on paper. (Or if you are using a batch system with punched cards and only get to submit it a few times a day. Ahem. Er. Was that me?) $\endgroup$
    – user737
    Jun 14, 2017 at 23:56
  • $\begingroup$ @nocomprende to be honest, I don't know how much you can "play around the edges" if the problem statement is "give a procedure to insert an element into a BST"... $\endgroup$ Jun 15, 2017 at 17:07
5
$\begingroup$

Depends on the course. For more theoretical CS courses, paper and pencil would be fine. But for the more "hands on" language courses that involve coding, a computer-based exam would be best so the students can concentrate on the thinking part of creating code and not on memorizing all the syntax nuances. There are too many in modern languages for anyone to remember. If all that is expected is to write "pseudocode" to demonstrate knowledge of an algorithm or design pattern, paper and pencil would be fine.

$\endgroup$
1
  • 1
    $\begingroup$ Welcome to CSE! $\endgroup$
    – Ben I.
    Jun 16, 2017 at 19:59
4
$\begingroup$

Some students (like me and my teenager) write slowly and almost unreadably, so nowadays it is common to allow them to use a keyboard for exams.

Computer based exams can be made better than paper, in terms of drilling down to the edge of the students understanding based on their answers.

$\endgroup$
1
  • 1
    $\begingroup$ I like to have students draw flowcharts at some points. This is very easy with a pencil. One student was disabled such that he could not use a pencil, but man, was he great at Visio! Of course, I made the exception during the test for this student. $\endgroup$
    – user737
    Jun 14, 2017 at 20:26
4
$\begingroup$

It depends.

As an undergraduate student, based on experience of exams that have been mostly paper based, I'd say:

For more 'theoretical' exam questions (describing algorithms, reasoning about time complexity, designing automata etc) paper is absolutely the way to go. It's the best way to handle mathematical questions and it's easy to clearly lay out an answer.

But this doesn't mean paper is a good way to assess knowledge of programming languages. I've had a few exam questions require me to write over a page of C or Java code, and these are just a pain. As an example, one question asked for a UML diagram to be converted into object oriented code. This would have been better assessed by asking "State the visibility of the field X on the entity Y?" or "Write a method signature for X" - having to write reams of boilerplate code in an exam is not a good use of time and isn't at all a good indicator of the candidate's ability to work in an actual software engineering role (which I assume is what's supposed to be being assessed, because it's most certainly not CS theory).

$\endgroup$
1
  • 1
    $\begingroup$ Welcome to CSE! I hope we hear more from you in the future. $\endgroup$
    – Ben I.
    Jun 19, 2017 at 12:41
4
$\begingroup$

I guess I’m late to the party, but I feel like there’s still a bit missing from the discussion. It’s too easy to say “CS != Programming” and get a round of applause. It’s also too easy to try to generalized from our own experiences: most of our students are not going to grow up to frequent csEducators.stackexchange. I think Winterborne, Ben I, and notquitezeus are right to start by thinking about who our students actually are, and what our course goals are.

Most of our students are at the introductory level, so let’s focus there. At my small, liberal arts college, the current intro CS student does not fit the stereotype of even a few years ago. Most are interested in something other than CS, but want to learn a bit of CS to be more effective at their other major. They are probably weaker on average than students who took CS when CS was a smaller discipline “only for the elite.” I for one, want to welcome these students to the CS world. My main course goals are:

  1. Give them a positive experience.
  2. Convince them that CS is everywhere, and empower them to use it.
  3. Encourage them to take even more CS. The intro course is not meant to be all-encompassing.

I try to avoid outdated teaching methodologies that historically would have pushed these newcomers out of the field (and, unfortunately, historically pushed out women and minorities in particular). I’m talking about things like macho programming challenges or extremely difficult labs that require all-nighters. For me, the jury is out regarding whether exams should be on paper or with a computer. I totally agree with Winterborne’s point that “Writing is not representative of their past experience in the course.” Many students hate the pencil and paper exams, don't feel empowered by them, and don't want to take more CS after the exam, running contrary to my course goals.

While the class focus is on algorithms and computational problem solving, rather than programming, exams still tend to be 10 questions of the form "write a function to compute ...". Many of the answers on this thread seem to go against IDEs, but the OP didn’t mention IDEs at all. I’m not convinced that an exam in something like IDLE would really give me less info than a paper exam. It would give me different info. By allowing students to actually run their code, an exam in IDLE would let me test their ability to quickly come back from a mistake, rather than brute force memorization skills. It would test their ability to understand and respond to error messages. Crucially, I believe working in IDLE on the exam (as they spend most of their time in the class, HW, and labs doing) would reduce test anxiety and stereotype threat, and this does line up with my course goals.

What are the downsides? Sure, error messages will point out their syntax errors so they can fix them. But that’s fine with me, because I’m testing ability to come up with algorithms, not knowledge of syntax. Exams meant to test syntax would run counter to my course goals. How about Ben’s example of confusing items and indices? This is certainly a common mistake. For some, it’s because they are rushed and sloppy. For others, it’s because they really don’t know the difference. I think that, even if they are getting help from error messages, I’ll detect the 2nd group of students. They are the same students who might not even realize a loop is required, and error messages are not going to help with that. Also, if Ben is concerned about detecting just how lost they are, he could have the computers record the history of error messages generated as students take the test, and could automatically browse it for particularly bad errors.

I'm not saying I necessarily think exams in IDLE are better than on paper. My colleagues are all against this idea. But I made a similar switch in Statistics and I'm convinced it was the right move. It allowed me to test students' abilities to synthesize resources I'd given them to quickly conduct a new applied data analysis. It traded rote memorization in return for an environment where students can showcase their skills and can go as in-depth as they want. CS is obviously different, but for me at least, this question deserves more thought.

$\endgroup$
3
  • $\begingroup$ P.S. I know that students could also "run their code" without IDLE. But, no matter how many times I tell them to “pretend to be the computer” and “write out what your code actually computes”, or make them do this in class, experience has shown that most will not do this on the exam. I'm open to suggestions for more conceptual test questions (at the college level), or for ways to really get the weaker students to actually catch their disasterous mistakes when they only have pencil and paper. $\endgroup$ Jun 19, 2017 at 18:05
  • $\begingroup$ P.P.S. we mostly hit our course goals via programming labs where students run large-scale simulations of real-world phenomena (often with some starter code). All credit to my colleagues for that course design. Unfortunately, exams remain 50% of the final course grade, and labs/HW don't really prepare students for them. I spend most of my in-class time on practice problems that are meant to be like exam problems, but students spend most of their time in front of IDLE on HW and labs. That's what got me thinking of changing assessment to most closely match student time expenditure. $\endgroup$ Jun 19, 2017 at 18:07
  • $\begingroup$ You've given me a lot to ponder here. I certainly agree wholeheartedly with the goals you've outlined, but I am not sure I agree with your conclusions - among other things, I don't find it difficult to discount syntactic mistakes entirely except for the occasional (very odd) edge case. $\endgroup$
    – Ben I.
    Jun 19, 2017 at 18:08
3
$\begingroup$

Our program teaches top-down design. This is an iterative process in which a text editor is indispensable. You stub in functions and methods, spec them out, and then write test code for them. After this, you begin writing code. If a method or function gets too complex, you refactor into reasonable-sized chunks.

This does not work well on paper. Stuff you put on paper is "dead."

Some of our assessment is paper and pencil; this is fine for the small stuff. A lot of our assessment comes in in the form of projects. For this, we assign complete projects of escalating complexity.

$\endgroup$
2
  • $\begingroup$ Isn't the question about tests, though? I don't imagine anyone is arguing that CS projects should be on paper. $\endgroup$
    – Ben I.
    Jun 20, 2017 at 22:08
  • $\begingroup$ My tests are lab practicals. These involve problems of escalating complexity that demonstrate various levels of proficiency from basic to advanced. I do not expect students to get 90% of this; I am looking for their level of proficiency demonstrated by what they can create. $\endgroup$ Jan 6, 2018 at 1:14
1
$\begingroup$

Save the trees

Requiring the exam be completed with pen and paper is a ridiculous anachronism.


  • If you are teaching something practical, like, say, Java 101, there is absolutely no point in asking a student to write Java code on paper. Nobody, not a single person, needs this. Why not use clay tablets if you are going for hard labor?

  • If you teach something like automata theory, then use a language capable of embedding proofs of what you teach, and require that students complete their tests by submitting proofs in that language. There is not a single reason why computer scientist wouldn't know how to use the most widely available form of computer as a tool in her work. And if proofs aren't possible or are too difficult, then use code that illustrates the solution. Anything other than that is "hot air"; feels unjust and discriminating due to the randomness of the opinion of the person responsible for verifying such test.


Some background info

I've studied CS and was briefly a TA. I had to submit exercises in Java written in pen on paper. I also saw the exercises submitted by other students. None had even a correct number of opening and closing brackets, let alone other typos.

My other background is in printing and typography. Having worked for a while in a publishing house, I assure you that the number of mistakes in mathematical and programming-related books is astronomical when compared to regular prose. It is not humanly possible to verify handwritten formulas in reasonable time. Every single textbook I had on subjects like math and CS had dozens of mistakes that I was able to discover, and the denser the material, the more mistakes there were.

There are prominent mathematicians like Vladimir Voevodsky who try to change this situation by developing better foundations for automating proofs. I believe that this is the way to go.

$\endgroup$
6
  • 1
    $\begingroup$ "Ridiculous anachronism" Is it safe to assume that you believe that paper should be banned in geometry and calculus classes as well? $\endgroup$
    – Ben I.
    Jun 19, 2017 at 12:32
  • $\begingroup$ @BenI. indeed so. Instead, students could use Geogebra, for example. There are great packages for symbolic math, s.a. Sage, or Maxyma (free ones) or Mathematica, Maple (paid-for). It's hard to imagine something so extravagant in undegrad, or even in graduate studies that wouldn't be covered by those. $\endgroup$
    – wvxvw
    Jun 19, 2017 at 12:47
  • $\begingroup$ Unfortunately, the kind of mindset which says "use pen and paper" pervades the offices of the ministry of education (at least where I live). Which is partly explained by incompetence, and partly by conservatism. None of which are commendable qualities for educators... nevertheless, instead of investing into improving the schools supplies, the funds are spent elsewhere :( $\endgroup$
    – wvxvw
    Jun 19, 2017 at 12:54
  • 1
    $\begingroup$ I can't say I agree with you. Computers don't necessarily make a great cognitive match with human brains, which are persistently analog. Our brains orient around spaces, faces, pleasures, and dangers, and learning is inherently social. I don't think that computers are a panacea, nor that they will ever fully replace analog processes. Use them where they obviously help, but be wary of falling into overly easy technologist thinking. $\endgroup$
    – Ben I.
    Jun 19, 2017 at 12:59
  • 1
    $\begingroup$ Re' the subject of this article: I want to see anyone using Sage or Maxyma at the same speed they can type arbitrary prose... this is not comparing apples to apples. And, surely, the article is not about testing, where different skills are needed... $\endgroup$
    – wvxvw
    Jun 19, 2017 at 13:10
1
$\begingroup$

Simply - yes, but not always. Paper-based programming exams make students think differently than they do in front of their editor. What's wrong with that? Nothing (as long as it isn't always that way). What's the benefit? Thinking differently.

$\endgroup$
1
$\begingroup$

One boring answer:

One big advantage of making students sit in a room in silence and write on a piece of paper is the best way of know of to ensure that the student has learnt the material, and is not paying someone to do all their work for them.

Of course this isn't a perfect system, but it is better than most things we have.

The question of why do those exams on paper instead of on a computer is often for boring reasons -- it is still hard to lock down a computer sufficiently to ensure no cheating, and also ensure it won't crash mid-exam. Paper has no such issues.

$\endgroup$
1
$\begingroup$

This thread is so long that I'm not sure that I picked up every point made, but I want to add an orthogonal thought that I missed if it is present.

Any timed exam, whatever the technology, old or new, will advantage some students and disadvantage others. Time itself can be the mind killer. I don't have a real answer for how to overcome it, other than to be constantly aware of it, and compromise when needed.

Your students are not like you. They are nothing like you. They are very little like one another, actually, with different learning modalities.

However, toward the end of my career I deemphasized formal exams (even finals when possible) in favor of papers and projects. Yes, I know that there are dangers with that as well, especially plagiarism.

$\endgroup$
-2
$\begingroup$

Yes. If you use a computer to take a test, for instance, you will have the tendency to code and check the program if the software is already there else you verify your answers on the Internet. As a skilled programmer, you can take tests manually and to the coding part you can shorten it. The school and college tests are just to ensure that you can remember the concept well.

$\endgroup$
1
  • 1
    $\begingroup$ Welcome to Computer Science educators! This answer doesn't exactly answer the question. The question asked about advantages of writing tests on paper, and you answer is addressing disadvantages of doing tests on computers. While that is related, it is not directly answering the question. Could you please edit your answer so that it addresses the advantages, so it fits the question? $\endgroup$
    – ItamarG3
    Jun 17, 2017 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.