How can we determine which students are a good fit for an educational program?

Question

tl;dr: How should I be thinking about assessing technical experience for admission into an educational program?

At my day job, I help run a program that invites undergraduate students to get "real world" experience working on a technical project with a team, under the mentorship of a professional software engineer. The program is currently 10 weeks long and involves working remotely on a team of 3 on a Java servlets-based web app.

So far, the program has been invite-only, but we're planning to open it up to the public. We're still thinking about ways to do that (might be fodder for a separate question), but in any case we're going to be in a situation where we have more candidates than we can accept.

The program is designed to support students with a relatively high experience level. They need to be comfortable coding in Java (including OOP, basic data structures, and tracing through an existing codebase), they need to pick up basic HTML and JavaScript, and they need to quickly grok client-server communications (e.g. get and post requests). Not to mention git, GitHub, Maven, workspace setup, team communication, etc.

We're not expecting any of our students to already be experts in any of the above. But they do need to be pretty comfortable asking questions, debugging, reading through documentation, and tracing through code. This is a virtual program where we don't get a lot of 1:1 coaching time with the students, so they need to be pretty self-motivated.

My question is: How can I determine which candidates will be a good fit for this program?

We've experimented with students assessing themselves as part of the application (e.g. on a scale of 1-5, how familiar with OOP are you?), but IMVHO this ends up not being a good indicator of actual experience level. We've thought about sending out some sort of assignment that they submit as part of their application, but to really test for everything we want, the assignment would be just as complicated as the real codebase we give them. Plus, what we're really trying to test for is a willingness to ask questions, time management, and enthusiasm more than pure technical experience.

With all of that in mind, the whole point of the program is to welcome students into computer science and encourage them to pursue a software engineering career, so in a perfect world, the assessment would come across as friendly and not too intimidating.

Sorry, this ended up being longer than I intended. I know this is a very broad question, but I'm looking for a general framework for how to think through this. I'd also be curious to learn more about how other groups are handling this.

I appreciate any insights into how I should be thinking about this problem.

Answer 1

tl;dr Approximate the "real world" practice/environment as the assessment.

Pre-setup

---

To begin with, I'm going to presume that there is some form or pre-screening, and/or qualification process. I.e.: having taken such & such course(s), being actively enrolled in some, possibly specific, institution, passed through an interview, etc. The process below most likely would be quite unwieldy with a couple hundred applicants, but should be reasonable with a few dozen. Especially if administered in batches.

In addition, as the program involves working remotely, I would expect that there is some form of real-time communication channel that will be utilized in the day-to-day working setup. I do not mean the telephone either. I could be as simple as using some form of I/M (AOL, Yahoo!, Messenger, etc.) or it could be an in-house developed and employed enterprise solution. Issues and comments on GitHub are good, but not sufficient when dealing with a mentor-ship program. Ideally, the solution (using different tools if necessary) needs to include a "private" channel between the mentor and mentee, and a "team" channel. A third channel for all the mentors and mentees to network, outside of the project development would also be beneficial.

Old tools, which I would recommend migrating from if in use, are Skype and TeamSpeak, though either would be better than nothing (provided you want to pay for TeamSpeak anyway). Two current tools I would suggest investigating are Keybase and Discord. Both are available for install on Windows, macOS, Linux, iPhone, and Android, as well as being usable form within a web browser. Both are also free, though Discord does have enhanced content available for a membership fee. Discord has voice channel capabilities, while Keybase does not. Keybase offers end-to-end encryption and has the ability to share, under encryption, files as well as having the ability to integrate with git and create an encrypted repository outside of GitHub. As a user of both, Keybase would be my choice for a professional development environment.

Assessment setup

---

I would suggest having the assessment as a collection of small projects. Adapted to the project and environment they will be working on, of course. Each assessment can target one or two skills rather than having a full-scale project to assess everything at once.

Projects could include:

• Create a routine to perform this task given parameters of X, Y, and Z and returning t and v.
• Given this pre-existing routine, extend it to handle this.
• This method, given (one, two, 15, false) should return expected_result, instead it returns bad_answer. Why? Fix-it.

In some of the assessments you can intentionally include ambiguous information, incomplete data, or other conditions which can only be handled by asking you (hence the need for real-time communication). Provided you've made it clear that questions are not only allowed, but encouraged and expected, they should be able to contact you with the questions. This also allows for checking how they deal with time management, if at some point you are "unavailable" and they don't get an answer. Do they plow ahead, developing based on a guess? Do they stop, waiting for an answer? Or, do they "table" that problem and work on another one?

If the work environment will include TDD or Agile, then the assessment should have the elements of that in some of the problems. As the work involves working as part of a team of 3, it would also be ideal if you could have the first few assessments deal with their own work, then have an assessment, or more, where three of the candidates, who may not even know each other, work on one project together. You could also have two candidates, and the third is a "ringer" who already works for the company.

As remote workers you will have no control over what resources they employ, including looking up how to write Java on Google. My opinion is that knowing how to find the answer is more important than knowing the answer. With those two points in mind, you could explain that "Google is your friend" or include links to the resources, including style guides, which most closely match your current standards, and encourage them to use those links in developing the "answers" to their assignments. This allows you to include ever harder issues, or more advanced concepts, and not only assess their knowledge, but their ability to expand that knowledge.

GitHub

---

You can also incorporate git and GitHub into the assessments, as early as the first one if you wish, by having them create their own GitHub account, provide an archive of the starting code if needed, and have each assessment be its own repository in their account. You can then use the issues and comments there to give feedback, even forking their repository, creating a new branch and making a pull request in their repository. One thing I would suggest not doing on GitHub is having a repository, or collection of repositories, from which the assessments, and code, are distributed. You could create a collection of separate accounts for yourself which are used with exactly one candidate at a time, and "reset" between as needed. With one account and all the repositories used in it, all the candidates would be able to see all the assessments, including ones they have not been assigned yet, and all the work of other, or prior, candidates.

Answer 2

Seems to me like you should have everyone submit a project within some theme you select every year (to prevent copying of previous projects). (This wouldn't be a general assignment, but more a sort of "show me something cool".)

Provide some boundaries - e.g., must be written in Java, must be submitted as a github repo, etc, that test some of what you need (you note in your question that you can't do everything - that's fine) - and then review the projects. I understand that for 1k submissions, this is probably pretty hard, but: this sort of thing will automatically weed it down to those who put in the time to make a project, and a bunch of these projects will probably be semi-easy to throw out - this project blew up when I tried to clone it? This project isn't even written in Java? This project has no OOP? This project is precisely two lines of code and one of them is a comment? And so forth.

It also tests exactly what you're looking for: the big thing in your question seems to me to be that the applicant is self-motivated and willing to learn. What better way to check that than an open-form project? In terms of the willingness to ask questions, that's the great thing about something like this, versus a test: google is open to the programmer. That probably sounds stupid, but if you're looking for a person who needs to be able to learn quickly and figure out problems for themselves, you very much don't want to test pre-existing knowledge like a test would, you want to give them something open form that you submit where google, SO, and other resources are accessible - it's a much better simulation of what they'd actually be doing.

The only real downside here is that this may come across as too intimidating. I think the key there is good phrasing in whatever the applicants see (the rules, or website, or what-have-you) - make it clear that you're not looking for the next Donald Knuth, but instead for someone willing to screw around, google, work, and put something kind of cool together. This may still be too much. However, I personally feel like it'd be a better approach than a more generic screening process.