Thing is, why keep Brees when you just drafted a QB so high?
It's funny how things work out. Team/management obviously had no confidence in Brees after the 2003 season, but he was still on the final year of his rookie deal so there wasn't a ton of harm keeping him around. Rivers' camp screwed up by holding out until late August, so they weren't going to start him at that point. In the old NFL CBA days, keeping Brees as an insurance policy wasn't the worst idea given how rough those rookie contract negotiations could get.
I don't mean to diminish the work that Brees put in during that offseason, but one thing that gets overlooked was that the Chargers overhauled their offensive line between 2003 to 2004. Oddly enough, Tomlinson had his most yards during the 2003 season with the makeshift offensive line.
2003: McIntosh-Garmon-Ball-Bogle-(revolving door)
2004: Oben-Fonoti-Hardwick-Goff-Olivea
Team got Roman Oben with one of the draft picks from the Manning trade. At that point he might have only been league average, but that was a huge step up from the previous year. Hardwick and Olivea were 2004 picks; Hardwick had a great career but Olivea kinda flamed out after a good rookie season. Fonoti had a weight problem but was pretty good in 2004. Four of the five guys started all 16 games in 2004. That continuity was a 180 from the previous season. Mix in 2004 being Antonio Gates' first full season and Brees was in a better situation to succeed.