Didn't teams go away from having one man be both the coach and GM over a decade ago? They said it was too much work for one guy and the biggest knock against it was that it had become impossible for a guy to coach his players up after he knocked them down as a GM in contract negotiations. Sutter was the last guy I remember doing this and it was already an outdated process when he started. That ended in disaster there too with him eventually going back to coaching which is what everyone always agreed he should have stayed doing in the first place. Quinn hired a GM above him in Toronto because he couldn't do both jobs. Eventually got fired by him.
If that's what has to happen to get a coach like Babcock maybe you do it. I don't know. I do think that's it's already proven to be another recipe for disaster though.
It doesn't have to be disastrous if done right. Set the GM titles like they are currently in Pittsburgh, where Botterill is "associate" GM. The Pens did that to ensure him the head job is his in a couple years. Rutherford is there to mentor him as much as he's there for GM duties.
Bring in an older, veteran GM (maybe Darcy Regier) and give Babcock "associate" GM status. For reasons you listed above, don't have Babcock involved in contract negotiations, arbitration hearings, etc. He would only have a hand in player evaluation, personnel decisions, drafting - basically nothing that threatens conflict of interest.
Maybe part of luring him in is developing a timeline, as I'm sure he doesn't want to coach forever. Maybe you give him 3 years to perform double duties and have him bring along an assistant he thinks is close to being ready for a head job, or you bring Nelson up as an assistant to be mentored by Babcock. I know Nelson doesn't sound interested in just being an assistant, but if you lay out a similar timeline for him to become a head coach, I'm willing to bet he goes for it. In 3 years, Babcock becomes GM/VP.