2016
Uncharted 4 - As a huge Uncharted fan, I was a little bit underwhelmed by this installment, but it's still a great game. Unparalleled visuals, good story and great characters. I was a bit let down with the set-pieces overall, but that's only because Uncharted 3 set that at an unrealistically high level. In a down year, it was my easy choice for GOTY.
HM: Firewatch - Never played it, but I don't feel like I have to with the type of game it is. I watched a playthrough of this on YouTube and was blown away at the storytelling and character development. Just a really atmospheric and well-made game.
2015
Witcher 3 - Looking back on this game (hell, I still play it actually), it's tough to make an argument against it as my favorite game of all-time. 2015 was a behemoth of a year for great games, but this one still easily stood out as my favorite. This ticks all of the boxes, with an amazing and vast open world, a great story, a staggering number of interesting and well-acted characters, and really good gameplay as well.
HM: Fallout 4 - Fallout 4 may have been a little disappointing relative to the hype, but it's still an unbelievably impressive and immersive experience. This is a Bethesda game by the books. Didn't do a lot to push the boundaries of open-world games, but it was still a very well-made game.
Bloodborne - This succeeded in introducing me as a Dark Souls noob to that type of game. It was just dripping in atmosphere and I had a really great time with it, even if I hated it at the same time. The enemy creations are second to none.
Undertale - This is another game that I never actually played, but from watching playthroughs of it, the writing stood out to me above all else. A lot of the dialogue in it is laugh-out-loud funny.
2014
Far Cry 4 -Very easily my "worst" game of the year choice. As a game, Far Cry 3 was straight up better. While it didn't do anything to up the ante on the Far Cry series, I still feel that the Far Cry series has near perfect mechanics for a first person shooter. It's just a blast to play.
HM: None. This was such a bad year for games IMO, at least of the ones I've played. I'm sure once I give something like Dark Souls 2 a try, it would take the title for me, but even then I hear that one is the worst of the series. Destiny was certainly fine, but it just felt so underwhelming.
2013
The Last of Us - This game gets so much praise, but it really does deserve it. Naughty Dog really upped their story-telling and character development from the Uncharted series, even if the Uncharted series already did that exceptionally well.
HM: BioShock Infinite - Probably not as good as the first one, but it was a very valiant effort. The ambition of the game alone is impressive. Amazing setting, top notch story, and solid from a gameplay perspective as well; the sky-rail thing may have been a bit gimmicky, but I still had a blast with it. Loved this game, and would have won my GOTY in most other years.
2012
Journey - The fact that I'm even considering a 2 hour game where (correct me if I'm wrong) you can't die says something. Quite honestly, there is nothing like it that I've ever played. The co-op was so well done, and the locations were beautiful and eerie. I don't know, listing out reasons makes it seem underwhelming because it's such an emotionally-driven game that is tough to define.
HM: Far Cry 3 - On a polar opposite note, Far Cry 3 was awesome. It is exactly what a first-person shooter should be. Awesome mechanics, great villain, good-enough story, fantastic and beautiful location. Had a blast with this game.
2011
Portal 2 - In a year that Skyrim set the bar for both open world games and games with replayability at an astronomically high level, I cannot ignore this ~10 hour game that has relatively little replayability. I challenge anyone to show me a better character than Stephen Merchant's Wheatley. He's so good in it that GLaDOS takes a backseat. This is the funniest, wittiest, most well-written game I've ever played, and it has the best mechanic of any game I've ever played (portal gun, of course).
HM: Skyrim - Skyrim drew the short straw on this one. This game did so much for introducing a fully realized world that is an absolute blast to explore. The replayability is almost kind of scary. Still holds up to this day.
Uncharted 3 - This one isn't remembered as fondly as most Uncharted games are for some reason. I'm a big advocate of story in games, but when you can play through a set-piece like a sinking cruise ship, I don't really care why I'm there as long as I can play through it. Hands down the best set-pieces in a game that I've ever played.
Side note: 2011 has to be the best year for games for me. Skyward Sword, although it has its flaws, doesn't get an HM for me. I thought LA Noire was fantastic as well.