I'm no expert, but my wife was told she should never play sport again after doing ligament damage to the same foot twice in a couple of years.Ligament damage in an ankle is a bigger concern than a broken bone long term.
So, from my small experience, I'd agree with you.