I find it difficult to predict success with the setup you describe. There are too many forces working against you. If your students are all residential and/or do most of their programming in labs that you provide then success might be possible, but otherwise, not likely.
- Students resist pairing and don't really know how to do it effectively. They don't understand what benefits it has until they see it work for themselves. They are unlikely to actually do it outside your view.
- Students find it difficult to find the time in their schedule to actually work together at one screen. Remote pairing tools are available, but not great, and probably work only for those already committed and experienced.
Making the project harder isn't really going to help you and it will frustrate at least some of your students. Letting them choose their own partners will encourage nothing new as they likely do some of that already but without real pairing. Also, letting them choose pairs doesn't get the real benefit of switching pairs and thereby spreading knowledge. Things happen outside your view, as you say. They will do as they wish and tell you what they think you want to hear.
Lecturing about pairing is especially ineffective. You have to show it and you have to let them experience it. Letting them opt-out gives you and them nothing at all. The people who want to work alone probably need the experience of working with someone else at least as much as the others.
Collecting feedback is, to me, the best part of your plan. But you should do that for all projects anyway.
I would do something like the following, though it will disrupt your class for a week.
Decide how many teams you should have so that there are 4-6 members per team. You choose the team captains. If the students don't know one another, then assign teams randomly. I used a deck of (handmade) cards with team names. People pick a card. That is their team. If they do know one another then "sandlot baseball" team choosing works well. In rotation, each captain picks their next team member from the remaining students in the class. This assures that there is a mix of abilities on each team and that teammates are "sort of" friendly without overemphasizing tribes and clans. Team "captains" have no management role. You can ask them to provide you feedback on how the team is doing and what it needs for success, but their job is not to tell people what to do.
Decide that you will spend a week of class time on pairing and kickstarting the project. Use a flipped classroom style for that week so that students read at home but work during class time. Other than reading there are no assignments during that week. You can spend a short amount of time at the end of each class period asking about questions on the reading. Don't start with this, though, or you won't know when to stop.
The first hour (or so) of the week in question is a demonstration, in which you and one of your students (maybe a captain), pair on the project itself. I would also incorporate test-first programming into this demonstration. Write no code until you have a failing test. Switch roles with your "partner". Switch partners also. Distribute any code you build to the class at the end so that they have a starting point. No code is written by any individual. Ever. For any reason. Just Say No.
If you really want the students to do this, there is one more aspect of agile development that you should adopt. Rather than having the students work from an overall narrative that informally specifies the project, or a formal specification that does so explicitly, give them the project as a "project backlog" (Scrum terminology). This is a list of cards that you have written that lay out the features of the project, one feature per card. The features are incompletely specified so you will need, as the project proceeds, to answer questions about what you really want. Each week you select a subset of the backlog for development that week.
The student teams work in the classroom, not at home for that entire week. Work hard on giving them feedback both on their product, but also on their process, as the week goes on. After one week you can probably turn them loose, but it would be good if you can supply them with a place in which they can continue to work as if they were in class, with the whole team (4-6) people there simultaneously. A cafeteria might work, but not a library unless it has work rooms used for small group discussion.
It is very useful if you can provide the class with an always-on communication channel in which they can always ask for and get help. This needs to be more than just email to/from you. A chatroom or a wiki works. So does an email distribution list to which everyone in the class can contribute. This makes the course 24 by 7, of course and you need to be available to give feedback. But the students should themselves feel empowered to answer questions that appear on the list. I didn't try to keep the teams from communicating with each other, even though the teams were working on the same project. Teams "copying" from one another was never an issue since information was basically free.
Note that I've used this successfully. Pairing, test-first, product backlogs, and small teams is a synergistic whole. But they need supervised practice or they won't see its benefit nor give it a fair trial. They also need a place and time to work together that won't disrupt their otherwise busy schedules.
Finally, make process feedback an important part of the project. You can ask for this to be given live or in writing. I found it useful to provide an anonymous feedback channel so that I could learn things that students wouldn't otherwise say.
