Yes, his skating is definitely a problem when it comes to his ability to be a high-end defensive player, but a lot of time it has a lot more to do with effort and/or awareness than speed.
You'll often see him gliding back in the neutral zone, and then an opposing skater passes him, and then he realizes he better catch up but he can't can't skate fast enough to do so. Some players simply have the defensive awareness and IQ to skate around naturally yet always be in the right spot. Others, like Boeser, ought to realize they need to bust their ass to get into a "failsafe" position on the ice so they can't get beat, because if they do get beat, it's too late.