One is only as strong as their weakest member.
If you went that route, the Hall of Fame would have well over 1,000 player inductees.
Tommy McCarthy was inducted as a player in 1945. McCarthy was worth about 16 WAR in his career. There are 1,000 players with 27 or more career WAR.
You don't compound clear errors arising from a capricious and faulty electoral system by considering them the new standard (or the old standard). You look at where the standards really are for the bulk of inductees.
For example, Lou Whitaker, who is a little bit above the junction between deserving and undeserving second basemen. I have him in the middle of a group of Hall of Famers - Gordon, Frisch, Sandberg, Biggio, and Alomar. The only second basemen I have ahead of him who are eligible are Ross Barnes (who stopped being a good player in 1876, when the National League changed the foul ball rules to prohibit bunts that bounced fair once and then went into foul territory) and Bobby Grich. I have him above a group of second basemen with decent but weakish cases (Billy Herman, Cupid Childs, Willie Randolph, Bobby Doerr, Tony Lazzeri, Hardy Richardson) and well ahead of players who were probably or clearly mistakes (McPhee, Nellie Fox, Evers, Schoendienst, and Mazeroski - they're intermixed with players who no one supports, like Tony Phillips, Lonny Frey, Bobby Avila, or Ray Durham). Lou Whitaker would not bring the standards of the Hall of Fame down, because he fits right within those standards alongside players no one considers to be mistakes and well ahead of players with marginal cases or no cases at all.
Bartolo Colon does not. I have him well outside the top 100 - in fact, he wasn't even on my original data sheet, so there are undoubtedly other pitchers who will get slotted ahead of him. Right now, I have him at #124, behind Mickey Lolich and Dolf Luque, and ahead of Fernando Valenzuela and Carl Mays. Good pitchers, sure, but not Hall of Famers. In fact, virtually all of the Hall of Famers behind him are pretty clearly errors - Herb Pennock, Lefty Gomez, Jack Chesbro, Addie Joss, Catfish Hunter, Jack Morris... I think Colon is about 50 players too far (about 75 current HOF pitchers, including those, like Smokey Joe Williams, who were prevented from playing in the major leagues because of racism, seems about correct.