The only reason I know that it's happened only once is the previous record for fewest fans at a major league game was six...set in Worcester when they played Troy on September 28, 1882.
By the way, for anybody wondering what the story behind the crowd of 6 was, both teams were lame duck teams who had been kicked out of the league for the following season and it was the last week of the season.
IIRC, the Orioles/White Sox closed door game was a matter of MLB making the Orioles play in Baltimore before the Baltimore PD was willing to allow fans to travel to attend the game (due to rioting). I think it was a case where game 1 of the series was moved to a doubleheader with game 3 (the first day the police would allow a game with fans in attendance), so game 2 (now game 1) had to be played in front of an empty ballpark, although some fans did watch through the outfield gates (the riots were pretty much over by then).