He played in an era where QB play in general started to improve drastically. I think it led to this weird thing where he had this reputation as this big clutch performer through his 2 Super Bowls, and then closed out his career with a few seasons that were statistically better than anything he'd done before on paper (4,000+ yards, 30 TDs, ~15ish INTs), but in reality, the standard of play for QBs was just getting better when he had those seasons.
Rank for QB rating each year of his career (From Pro Football Reference; I don't know what the cutoff is, but it seems like the fewest games played for QBs that are ranked is like 8-10ish starts per season, mostly looking at starters):
2004 - 55.4 (NR)
2005 - 75.9 (23rd)
2006 - 77.0 (18th)
2007 - 73.9 (25th)
2008 - 86.4 (15th)
2009 - 93.1 (11th)
2010 - 85.3 (17th)
2011 - 92.9 (7th)
2012 - 87.2 (14th)
2013 - 69.4 (35th)
2014 - 92.1 (15th)
2015 - 93.6 (13th)
2016 - 86.0 (22nd)
2017 - 80.4 (25th)
2018 - 92.4 (21st)
2019 - 82.6 (NR)
So by best finish by rank, you have:
7th
11th
13th
14th
15th
15th
17th
18th
21st
22nd
23rd
25th
25th
35th
His magnum opus of a season was 2011 when he put up his best year relative to his peers AND won the Super Bowl. He had over 4,900 yards, 29-16 for TD/INT and was 9-7. I think most can agree that was a great year. But other than that, you have another decent year in 2009 (albeit not top 10) and then nothing but middling to awful seasons where it was split on whether he'd be a top 20 QB in the NFL (at the time) or not.
He had 2 Super Bowl runs, which is obviously 100% what his positive legacy is, but lost the 1st game of the playoffs the other 4 times he played in the playoffs.
I honestly think his HOF is absolutely absurd, but I fully expect him to get in.