Castle is one of my most played defenders. He's pretty good as he is, most people just don't know how to play him well.
His barriers are meant to delay and cause attackers to waste their breaching charges or rounds for their special weapons. Sledge really is the only one that can tear right through them with ease and not every team has a Sledge available. Everybody else has to hit the barrier 20 times to break it without a breaching charge.
I've lost a game being the only attacker left and the room was sealed with Castle barriers and I had no more breaching charges and ran out of time trying to break the barrier. I've also won a game that same way. I saved one barrier and there was a minute left and I knew the attacker was going to go through a certain door and put up the Castle barrier and they couldn't get through and ran out of time.
You can make a similar argument about every single character. I'm not trying to discount your experience but in my experience, when viewed relative to what other defenders are bringing to the table, Castle could use a buff.
Breaching charges are not a rare commodity. Also, Ash can absolutely blast through with ease akin to Sledge. 3 breaching charges is a lot. Those are 3 breaching charges which are accessible to a very high percentage of attacking operators. Like someone else said, I have an Ash or a Sledge on my team as an attacker probably 95% of the time I play ranked. I'd say I have both of them on the team at the same time a high amount as well (although, full disclosure, I do main Ash/Thermite/Thatcher myself). Than you end up with other operators running breach charges as well (Thatcher, Twitch, Blackbeard etc.). That's a lot of breaching charges running around.
My point being that I do not think, in my experience playing Ranked and Casual, I've ever found myself in a situation where Castle's barriers are still standing with no one left to breach them down. Has it happened before? Of course. That being said, I would really be interested in seeing how much play he's getting in ranked where literally no half decent team should arrive in a situation where they have to manually punch out his barricades. Geez, he's not even a direct counter to Fuze right now which would be a huge plus and maybe make him somewhat of a Mute alternate.