Height means a lot when it comes to recruiting in baseball. As someone once said, "A small man has to prove he can play. A big man has to prove he can't play."
I'm not disagreeing it's not a factor, but not as large of a factor as the NBA for instance. Also with 1200+ players drafted (40 rounds) every year, short guys at least have a chance of being drafted and a chance to show their worth. Advanced stats in baseball have really helped smaller guys get at least a 2nd look compared to ~20 years ago when there were scouts sniffing around the local high school games for talent (not for me, I did track).
Every A level game I goto I'm commenting on how small the players look. Spurgeon on skates is taller than half of the players on the field. After team warmups the tall guys go to the bullpen and the short guys go to the dugout.