tl;dr: How should I be thinking about assessing technical experience for admission into an educational program?
At my day job, I help run a program that invites undergraduate students to get "real world" experience working on a technical project with a team, under the mentorship of a professional software engineer. The program is currently 10 weeks long and involves working remotely on a team of 3 on a Java servlets-based web app.
So far, the program has been invite-only, but we're planning to open it up to the public. We're still thinking about ways to do that (might be fodder for a separate question), but in any case we're going to be in a situation where we have more candidates than we can accept.
We're not expecting any of our students to already be experts in any of the above. But they do need to be pretty comfortable asking questions, debugging, reading through documentation, and tracing through code. This is a virtual program where we don't get a lot of 1:1 coaching time with the students, so they need to be pretty self-motivated.
My question is: How can I determine which candidates will be a good fit for this program?
We've experimented with students assessing themselves as part of the application (e.g. on a scale of 1-5, how familiar with OOP are you?), but IMVHO this ends up not being a good indicator of actual experience level. We've thought about sending out some sort of assignment that they submit as part of their application, but to really test for everything we want, the assignment would be just as complicated as the real codebase we give them. Plus, what we're really trying to test for is a willingness to ask questions, time management, and enthusiasm more than pure technical experience.
With all of that in mind, the whole point of the program is to welcome students into computer science and encourage them to pursue a software engineering career, so in a perfect world, the assessment would come across as friendly and not too intimidating.
Sorry, this ended up being longer than I intended. I know this is a very broad question, but I'm looking for a general framework for how to think through this. I'd also be curious to learn more about how other groups are handling this.
I appreciate any insights into how I should be thinking about this problem.