That's what the Veterans Committee does, and has given us luminaries like Harold Baines and Chick Hafey. The Frankie Frisch Superfriends era inducted probably half of the worst choices all by themselves.
The reason the BBWAA has the vote is because the Hall of Fame was designed to be a tourist trap and the actual election process was an afterthought. The BBWAA was at the time the only organized body of professional baseball experts in the world - there were no baseball historians, there was no player's union, there wasn't even a statistical guide that voters could easily access.
Simply giving the vote to "the players" probably would have ended up with really bizarre choices like Hal Chase or Fred Merkle in the Hall, because, quite frankly, the players don't really have a better handle on who is actually great and who isn't than the writers do (which I think the VC selections have shown).
