Speaking to former Corrie star Nicola Thorp and telly host Matthew Wright about the issue, X Factor host Dermot compared Simon to "Mariah Carey".
He said: "He's awful at timekeeping. Mariah stages."
Having worked with him in the past too, Rochelle explained: "We're talking five [hours late]. It could be hours and hours."
Dermot teased: "In many ways it's Cowell Time – somewhere in the mid-Atlantic that only he seems to operate in and everyone else seems to go around him."
However the new dad confessed he isn't much better, as he added: "My wife calls it O'Leary Time. I'm not great."
The conversation on This Morning came after Ant and Dec revealed they secretly tried to quit Britain's Got Talent back in 2012.
Writing in their new autobiography Once Upon A Tyne, Dec, 44, said: "A couple of hours after we arrive at the theatre, we focus on one of the key elements of any BGT audition day: waiting for Simon Cowell to turn up.
"Once we've got started on that, we'll often film interviews with some of that day's acts, then have a bit of food, then get back to waiting for Simon."
Simon, 60, also contributed to the book and admitted: "They used to have a real issue with me being late and it used to drive them nuts."
But Ant added: "We've still got a real issue with him being late and it still drives us nuts."
Simon won't be joining the Britain's Got Talent live finals this year as he is staying in the US due to ongoing restrictions with the coronavirus pandemic.
Instead, he will be replaced by former BGT winner Ashley Banjo, who won the show with his dance troupe Diversity.
This Morning airs weekdays at 10am on ITV