I think it's because fans only look at the upside of all prospects. EG, Poulin looks to be a sure fire top 6er. Legare looks to have good upside to be the same. Addison is going to be a top 4 PMD who can QB a PP. They look at all of this and think that a "rebuild" could be done as quickly as 3-4 seasons.
GMs on the other hand know how hard it is to acquire talent and that more often then not prospects bust, picks don't pan out and that a rebuild that's planned for 4-5 years, could in fact take 8-10 years (Edmonton) or 7-8 years (Buffalo). Just because a GM hopes that it'll be quick, frequently it doesn't actually workout that way. Sure some teams have managed to do quick re-tools but it's rare that those same teams are also highly successful over extended periods of time. So while fans may want a GM to rebuild, a GM knows that he very easily might not last the rebuild and that the fans who say they want one could very easily get fed up with it if it takes longer then what they think it should.