Yeah that would be sticky... that max $8.1 million per season amount (to just have to give a 1st, 2nd and 3rd as compensation) would be enough for Dubas to have to sit and think; Nylander is almost certainly worth more than just those picks on paper (with Nashville that 1st is likely to be pick 25-30, so... meh), but $8.1 million is pushing it, and is probably a million and a half more than what Nylander is actually with. I think ultimately Dubas matches at that amount, but that would be interesting to see. Now if a team offers above that threshold into the $8.1 million to $10.1 million range for Nylander (and compensation is then 2 firsts, a second and a third for the Leafs) then I think Dubas lets Nylander walk (and regardless of how much Nylander likes Toronto and the team I can't imagine he would turn down such a contract), but that would be a gross over-payment for Nylander; he is young a top-line winger and potentially could make the move to being a top-6 centre full-time but he isn't worth $9-10 million, not even close.
An eleite team (with the cap space) offer-sheeting at $8,118,641 is not necessarily a bad idea; it would test Toronto's resolve, and if they matched, force them to overpay a bit for Nylander making them more vulnerable next off-season when Marner and Matthews are up, and if Toronto didn't match then $8,118,641 isn't so much of an overpayment that it would be unpalatable given Nylander already is a two-time 60 point guy (in his first two seasons) who is just 21 and will likely continue to improve for a couple seasons more. Furthermore, for a contender those picks they would lose in compensation to Toronto aren't likely to be all that valuable (obviously a bottom-feeder team wouldn't offer-sheet Nylander because they would risk losing a first-overall pick).
What is holding this scenario back is that there needs to be a team who is a contender (so those forfeited picks are low) who has a pressing need for a player with Nylander's skill-set and can throw around $8,118,641+ to offer-sheet Nylander and still have enough cap-space left to keep their own players under contract. That hypothetical team having a vested interest in weakening the Leafs (eg. a division or conference rival) wouldn't hurt as motivation for the offer-sheet either. The fact that there doesn't seem to be a lot of teams out there that tick all these boxes is why we haven't seen an offer-sheet yet, and why offer-sheets in general are so rare.