The Destroyer armor is made of Uru. And Uru, no doubt, is very durable, but nowhere near as durable as Adamantium or Vibranium. What makes Uru strong enough to rival Adamantium or Vibranium (I said rival, not tougher), is it's ability to soak enchantment, and it gets a lot more durable once enchanted, like Mjolnir. There was also a description in the Marvel database where Thor lends some of his magic to make Mjolnir even more durable.

In Avengers 51, Thor was seen punching an empty Destroyer Armor, and he was making dents on it, while at the same time, his hands were noticeably bleeding. This particular Destroyer armor may or may not be enchanted, since it is being used as a mere training equipment. But an inanimate Destroyer armor, being used as a punching bag, is without question, not as durable as a Destroyer armor being inhabited by Odin's spirit. Considering Thor has carved uru to make a makeshift hammer with only his finger, making a few dents on uru is not an impressive feat.