Sooo …there are two possible explanations:
Either Link is somehow able to magically resize everything he owns so it fits him perfectly. This would also explain why oversized weapons like Lynel Crushers suddenly shrink to a convenient size once Link picks them up, but if it applies to armor as well, then it should not have been such a big deal for Muzu that the Zora Armor “fits Link perfectly”. If the armor resizing thing is canon lore-wise, then it would have ruined an important key moment in one of the main quests.
… but if the armor resizing ability is NOT canon, then why did the Warm Doublet fit Link at all? I guess the “Old Man” made the Warm Doublet specifically for Link and just pretended it was his own shirt, hoping that the little guy doesn’t ask too many questions.
… or the old man once wasn’t a big man, but a small boy, and it’s his old shirt from those days?