You're confusing defense with defensemen. Goaltenders are definitely a member of the defense.
Well goaltender isn't really part of the defense overall, but forwards absolutely are. Goals against is a factor influenced by both goalies and players, more so goalies, but players have an effect thus I still factored them in for my calculations. Besides goalies technically have an effect on shots metrics as well just saying (Rebound control, puck moving, puck tracking, etc.), so you might as well say you cannot measure true defense with just stats.
Either way you're correct that people confuse defense with defenseman too often. I constantly have to tell people that Nashville does not have the league's best defense even if they have the league's best defenseman (And even then that's debatable, it's more so their top 4 that's the undisputed best than their entire blueline).