As with most battles involving the Destroyer, it depends on whether or not Ego has the capacity to damage it. I would say not, classically, but these days who knows. Even so, if the Destroyer starts delivering planet-splitting hits, Ego's going to be channelling his power to his defence more so than offence.

Ego has the power to shatter planets and stars (IIRC), so flip a coin, the writer can make either win.

If we were talking about the Destroyer at its best, I would give it heavy odds. If we're talking modern Destroyer, where it was said that Odin-level beings could 'shred it like tinfoil', then maybe not.

Like many beings introduced way back and which had a high starting point, the Destroyer has gone through eras where it was seemingly invincible, was over-used, had low points, and eventually was mainstreamed into an above-Thor but below-Odin type threat.