As an Ivy League grad, here's my perspective: part of it is marketing/branding and part of it is not. However, that effect is very real and very relevant so it shouldn't be discounted. I definitely agree with your general point that it's what someone does with their degree that matters, not where they received it, but I can tell you from personal experience that my diploma alone has opened doors and raised eyebrows for me that my GPA never, ever could have. In general, it's the graduate/professional programs that really make the Ivies the Ivies. From an endowment and research perspective, their records speak for themselves.
I'm proud of the fact that I got into and graduated from one of the most difficult universities in the world. Was it worth it? Other than from a personal relationship standpoint, absolutely not. I nearly died there multiple times and my current career is different from what I majored in back in college. My wife and I came from the same school, but you can bet your ass that my kid isn't following suit. No way is my kid going to be sent into that meat grinder.
For one, for out-of-state students there is likely not a huge difference in cost between UMich, UVA and the Ivies; private school in America is a rip-off nationwide. For two, it's an apples-to-oranges comparison because (and I generally don't mean this to be insulting) thousands of students currently attending either of those universities unfortunately would have no chance to get into Harvard, Yale or Princeton in the first place whether deserved or not.