The podcast, which is a candid look at dealing with cancer and all that it entails, launched in 2018 and was hosted by James, Bland and Lauren Mahon.
Mahon and Bland's husband Steve continued without them but have now said they are ready to "hang up their headphones" and pass it on to someone else.
Steve told BBC Breakfast that he wanted it to continue without them as it is so important, but he admitted it had taken a "toll".
"It's hard to kind of keep talking about this stuff – particularly when we've had to deal with obviously Rachael and Deborah over the last few years," he said.
"There aren't many people doing podcasts where two people have died from the subject matter."
"It's a tough thing to keep talking about," he added.
Mahon - who has now been clear of cancer for five years – told BBC North West Tonight: "I have struggled over the last year getting back in that studio without their seats being filled."
"How do we get back into that studio without these two remarkable women that began this podcast?" she added.
Mahon said that people are getting diagnosed everyday and that the world of cancer changes.
So she said she thought it would be "more appropriate and probably more poignant and relevant" if people going through it now took over the podcast.
Watch: Dame Deborah James' posthumous book reaches number one