But rather than Simon's tardiness, it was Dermot's lack of screen time that gave him the hump following his sacking from the show in 2015.
Dermot said: "I've had my state-of-the-union meeting with Cowell as well. You're watching the show one day and going, 'I'm not in any of this.'"
However the popular presenter explained that Simon manages to diffuse difficult situations straight away and gave an example of an offer of chicken pie.
The dad-of-one added to The Sun: "The most delicious chicken pie you've had in your life turns up and you have this things and you go, 'I don't think I'm in the show enough.'
"He goes, 'Yeah, I totally agree. Yeah, absolutely.'
"Then you go, 'Great, now I can just have a great time,' because he's great company."
Dermot revealed that Simon did stick to his word too when speaking to fans during an interactive talk for Ant and Dec's autobiography.
He said: "To be fair, he's a man of his word. When it happened to me, I was in it more. And I felt more part of the team.
"He's a great night out as well, terrific company. When you see that little sparkle and the gossip starts, he's a world of fun. I just wish he wasn't so late all the time."
Dermot had other grievances though as, in 2011, he made it clear he wasn't very happy about being passed over for the hosting job of The X Factor USA.
Simon went on to claim that him being dropped from the UK version in 2015 was a "misunderstanding" and Dermot was invited back a year later.
Ant and Dec wrote in their book that, back in 2012, they tried to walk away from Britain's Got Talent.
They wrote that Simon was often late to filming and that they felt sidelined by the telly boss.
Luckily for fans, he managed to sweet-talk them and get them back on side, and they've hosted ever since.