Two answers are possible: a direct answer to the explicit question(s), and a response to the implications for education hinted at by the excerpt.
The written questions
When does excessive collaboration become plagiarism?
It never can.
[W]here should the line between excessive collaboration and plagiarism be drawn?
Such a line cannot exist.
As ctrl-alt-delor covered in his answer, plagiarism is claiming the words/ideas of someone else as your own. To verify that I searched, and read, many definitions from dictionaries, online references, academic and professional sources. The summation of all those sources is that using the material, or work, of someone else and presenting it as your own is the primary factor, or offense, of plagiarism. The nearest I could find for an official definition of plagiarism was in a fact sheet from the Office of Science and Technology Policy, in 1999:
Plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit, including those obtained through confidential review of others' research proposals and manuscripts.
Source: Executive Office of the President, Office of Science and Technology Policy, Fact Sheet, Oct. 14, 1999
After review, and an appropriate comment period, it was finally edited to read:
Plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit. Research misconduct does not include honest error or differences of opinion.
Source: Federal Register Vol. 65, No. 235, Dec. 6, 2000, pg 76262
I did find one good academic source:
Plagiarism is presenting someone else’s work or ideas as your own, with or without their consent, by incorporating it into your work without full acknowledgement.
Source: University of Oxford, Plagiarism
Their discussion continues with the comment:
The necessity to acknowledge others’ work or ideas applies not only to text, but also to other media, such as computer code, illustrations, graphs etc.
The question places collaboration and plagiarism as possible coexistent concepts. They are not, as recognized in a comment by mickeyf. If two (or more) students are collaborating (working together) on a project, neither can plagiarize the other's work. "There is no I in team". [The excerpt also mentions using online resources and relying heavily on peer code, both of which could be plagiarism if not acknowledged.] It is possible that the habit of doing very little work in collaboration with others could lead to plagiarism, making the student at-risk of plagiarizing, as the excerpt noted.
It is possible, in very rigid, or high-stakes, environments, to be morally obligated to cite sources of inspiration, such as from casual interactions with others, to acknowledge their input. The U.S. Department of Health & Human Services, The Office of Research Integrity, in their section on plagiarism, Acknowledging the Source of Our Ideas, covers some research published by Alan Gilchrist in a 1979 Scientific American article where Mr. Gilchrist purportedly felt obligated to credit a source outside of the normal research he conducted. Holding students to that standard in the classroom would require them to credit the instructor, the textbook, and likely half of their classmates, in every program they wrote. It would be interesting to see a class project with a comments section listing sources that was longer than the code itself.
The objective of the classroom experience ought to be learning, the instilling of knowledge, habits, and patterns of thought, into the minds of the students. When collaboration is part of that process, there is no need to be looking for the evils of plagiarism where it cannot exist. When, however, the excessive collaboration, or other impediments, interfere with the process, then they need to be addressed. That, however, is unlikely to be the result of any bright-line test or standard. Rather, it is likely to come from the instructor's observation of patterns of behavior, and work, exhibited by the students affected.
The educational implications
To begin with, the excerpt seems to be the rationale (or sales pitch) for the authors' new tool, TMOSS, rather than a warning to instructors, or an indictment of students.
The authors give excessive collaboration as one factor that could identify students at-risk of plagiarizing. They neither claim that it is plagiarism, nor that it is the only factor that leads to being at-risk. The apparent objective is to identify students in need of additional help in the class, not to weed out or identify plagiarists: "it is hard to know which students need extra help," and "so we can intervene and provide additional class resources to help them succeed."[ibid.]
The implications are that when students become more dependent on the assistance of others, and outside sources, for results, they develop malformed habits that reduce their learning, and perhaps affect their future work as well. Prefaced with the presumption that the classroom experience is intended to "educate" the students, finding those that might need extra help using any tool available can be helpful. If noticing that a student is engaging in excessive collaboration (though I have no clue what level is considered excessive) can help to alert the instructor to a student that needs additional help, or needs guidance in study/work habits, it may be worth considering that approach. It is possible, however, that students who engage in excessive collaboration are developing work habits which will lead to greater success in software development, especially in Open Source development, than their peers. Teaching the students how to learn, not how to work, seems more appropriate to the educational system, including Computer Science curricula. Teaching them how to work seems more appropriate in vocational training or rehabilitation.