Yes, many prospects both big and small in stature never make it to the NHL, that is why size if only one variable to look at and most likely a secondary or tertiary variable in terms of priority. The majority of the best players in the league range from 5'11 to 6'3, so only when prospects are outside that range does it become important to look at. The risk of failure increases in each deviation from the norm. In terms of risk 5'6 > 5'7 > 5'8 > 5'9 > 5'10 and on the other side 6'4 < 6'5 < 6'6 < 6'7 < 6'8. A 5'8 player and a 6'6 player should be evaluated in a similar risk rating; there aren't very many players that size that make it to the NHL and likely a similar percentage make it at each height. The risks and reasons for not making it are different but the amount of risk is comparable.
The factors that lead to the success of prospects include skating, skill, hockey IQ, compete level, character/work ethic and physical attributes/abilities. Being overly focused on size can result in overlooking other very relevant factors. I also have a preference for a bigger more physical team but it is exactly that, a preference. It isn't an absolute rule. I wouldn't pass on drafting players like Brad Marchand, Brendan Gallagher and Darcy Tucker simply because they are undersized.
The other factor you need to consider is that the role of a GM is asset management. Subjectively you may not value smaller players but if you were a GM other GM's would. Even if you wanted to trade every small player away the goal would be to maximize the return for the asset. Your trade ideas suggest throwing prospects of high market value away. Your preference for bigger players blinds you to getting the best return. Abramov and Brannstrom were key pieces used to acquire Duchene and Stone. Trading Abramov and the 32nd pick to move up to 28 is a massive overpayment. Trading Brannstrom for a first in the mid 20's is also a massive overpayment.
Aside from that the majority of our young players/prospects are 6'1 or taller. Norris, White, Tkachuk, Formenton, Batherson, Brown, Paul, etc are all tall. We won't be a small team if we happen to draft a few players under 6 ft.