Simple demonstration:
Presenting this and asking, "What do you see?" is sure to get a variety of answers. Most might be along the line of a bird feeding baby birds. Not all will be, however. Possible answers include "A bunch of birds," "A birds' nest," "Tiny leaves," "A pretty picture," and even "A fuzzy background." None are wrong, and all will show what the answerer first focused on as an answer to the question. None of the answers reflect what else they might have noticed, given more time, or what else they could say, if asked in detail about it. The fact that you wanted to know the species of the bird is not in the question, so not necessarily in the answers.
Students have a particular mindset for taking tests, or other evaluations, that contributes to the problem. Students are driven to maximize their time, and the potential for high marks. They will do anything they can to find the answer in the shortest possible time-span. Once they have a possible solution, they will test it just enough to see if it is probably the answer, do whatever is required to demonstrate that answer, and move to the next question.
You can attempt to precondition the students to be thinking in the right mindset to "see" what you want them to see by ordering questions in a fashion that leads the student to the proper mindset. There is no certainty that such a tactic will succeed, however. Many students, especially in the higher levels of education have a test taking skill where the test is scanned first, then questions are answered. Shorter and/or easier questions are answered first, longer questions latter, and the hardest ones last. In that style of test taking the preconditioning questions can be somewhat removed in time from the final target and may have lost their "conditioning" value.
Asking observational questions in tests, or quizzes, will result in answers showing what the student focused on the fastest that might answer the question. The answers are not even a true measure of how the student thinks, or how they would evaluate the same information in a non-test environment. The more obvious clues you can give about the expected answer the greater the odds are that answers will fall closer to your expectations. Yet, even that is of limited value in most programming contexts. Even with the clue of "design problem" the students, as seen in your case, are going to branch off into several directions, only one of which is going to be how the given code is connected to the greater whole. As seen by the answers you did receive, other possible branches are the "design" of the code inside the class, and even error checking, which is often considered a "design" issue.
Unless a code segment has a compile-time error, or other syntactical error, a blanket statement of "find the bug" will lead to a range of answers, most of which are technically correct while not being what you had in mind. If it is a syntax issue, or an error which will cause a compile-time error, the question should still be explicit in that a syntax error, or compiler error is expected and you want the cause, or the correction to avoid it. Without that qualifier it is still often possible to find a "problem" with the code that is other than what you were expecting.
Since I don't know Java well enough to be sure, I'm only making a stab at a possible re-work of the question. The principles should carry over, even if the technical details do not, however.
In the question cover the facts that, first, the comments state, There must be only one object of this class
, and second, some future coder using the class might not read the comments. State that you want them to make the needed changes "at the design level" which would force the future coder to only have one object of that class. "The best method will cause a compile-time error if the rule is violated." An alternative statement could be to rewrite the code such that "it is impossible for two objects of the class to be instantiated by the program."
That's the part I'm not solid on. I don't know that a compile-time error would be created in a program trying to create two objects of that class if you turn it into a singleton class using the changes you described.
If you have taught, and used, the singleton pattern, then they should have a decent change of coming to the conclusion that such is what you want. Still, other solutions are possible, such as a wrapper class that tests for how many there are already, or some kind of internal test for prior instances. The more specific you can be to eliminate alternative "fixes" the better.
My opinion is that in a testing environment it is unwise to ask for evaluation of designs in search of improvements. As I covered earlier, this will not offer any reliable insight into the abilities or thinking patterns of the tested students, but rather it might offer insights into their test-taking patterns. Since I doubt the latter is your objective, the tools used might be better replaced. It might be better to tell them to use a pattern to change (or even create) something, and the skeleton of that can even be bare bones. To determine that they understand the singleton pattern, as you taught it, you could instruct them to convert the following skeleton into a singleton class:
public class UserSystem {
private type var; // instance variables
public UserSystem() {
// instantiation code
}
public void methodOne(argList) {
// methodOne code
}
}
There are no "bugs" or design issues to correct. What the class is, or does, has no bearing on how it can be made into a singleton class, and is therefore, unimportant to the question, and only distracting to the student.
There is one other thing you can do before the tests, preferably at the beginning of the course, maybe even include in the sylubus. You can state that you will use certain, defined and provided, phrases to emphasize what elements of the material you are examining. This may, or may not, be possible in your situation, but at the very least you can develop and publish, a rubric, for the tests before they given to the students. This allows them to see what you're testing for, and evaluating against. They can then focus their attention on the details that matter. If you're not testing their ability to find syntax error in provided code, you can say so, and even tell them to ignore any they find and presume that it is correct syntax for the purposes of the question. After that instruction, any answer that is about the syntax is obviously wrong, even if they did find such an error and use it as the answer. You can even add such statements, where warranted, to the body of a given question. Though, if on the question itself, I'd make it a bold and separate line in the text of the question.