Family is a very complicated thing. As others have said all you can do is what you can do, and from there you need to protect yourself and your growing immediate family mentally.
My situation is a bit different. My parents divorced when I was 18 months old and I didn’t have a great relationship with my dad, in large part because of my stepmother. They have 3 kids together and I’ve never felt like I fit in to their family…but I was forced to for the majority of my childhood and that does a number on a kid.
As I’ve gotten older and “done the work”, I’ve realized that I just need to avoid that situation as much as possible for my own sanity. Oddly enough, at the same time, my dad has been very into trying to have a better relationship with me. On the one hand, he’s my dad and on the other hand, I’ve done a LOT of work through therapy and self reflection to work through shit from my childhood and I just don’t want to open that door again at 40. You had 37 years to build a post-divorce relationship with me, your wife was awful to me as a child and continues to be spiteful as an adult and you were either too clueless or too afraid of confrontation to do anything about it…I’m set.
My husband and I are both only children and each have one parent in the picture (my mother in law passed about 6 years ago). We’ve built a network of friends that are our family. Sure, I get a bit wistful when I read about extended family holiday gatherings because that’s not something we have, but we don’t lack for love and support.
Moral of the story, do whatever you need to do in order to protect your peace, your wife’s peace, and your munchkin’s peace. Family is what you make it.