Simon has been forced to step back from the show after breaking his back from falling off an electric bike, as he has been recovering at home in Los Angeles.
Piers has already claimed that he was approached to fill in for Simon on the panel but had turned it down because of his feud with David.
Whilst details of their spat have never been made public, it seems the feeling is mutual between them, as David reportedly threw a strop at the thought of working with Piers.
A source told The Mail on Sunday: "I heard that David threw his toys out of the pram.
"Imagine David and Piers sitting on either end of that judging panel – what great TV it would have been."
Piers sparked rumours about their feud in his column for the Mail, writing: "I was quickly sounded out to see if I was potentially available but had to explain that if I was confined to the same close quarters as that insufferable twerp David Walliams, it wouldn't end well."
The stars reportedly fell out after Piers realised David had "liked" a number of negative comments about him on social media.
Piers was a judge on the show from 2007 and was replaced by comedian Michael McIntyre for one season, and later Alesha Dixon up to the present series.
A BGT spokesperson said about the reported casting process: "The decision regarding who would step in as guest judge was made solely by the producers and Simon with Ashley Banjo the top choice to take part."
BGT returned last night with a huge musical number from Ant and Dec, and saw comedian Steve Royle become the first contestant to go through to the live final.
The show continues next week at 8pm on ITV.