He said he demanded Picower return all the money he had withdrawn from his Madoff accounts over the years.
An attorney for the Picower family calls Madoff's allegation "entirely false."
"There were no conversations between Mr. Madoff and Mr. Picower after Madoff's arrest," said Marcy Ressler Harris, a partner at Schulte, Roth & Zabel in New York.
(Read More: Madoff, in Christmas Eve Letter, Says Insider Trading Has Gone on 'Forever ')
In a lawsuit filed in May, 2009, Picard alleged Picower was "one of a handful of BLMIS clients with special access to information," including Madoff's fictitious rates of return, and that Picower knew Madoff had backdated trades for him.
Picower died five months after the lawsuit was filed, and his wife settled with Picard in 2011. But Picard said Madoff had nothing to do with it, or any of the other money recovered. Picard said even when he interviewed Madoff in prison on two separate occasions, Madoff was not forthcoming.
"The interviews of him were, really, to confirm the information we found, because those interviews took place shortly before some of our litigations began, and we put this case together really from a blank slate," Picard told CNBC.
Madoff wrote that he has repeatedly offered more help to Picard and his team, but they have not taken him up on it.
"You should be aware that I have continued to offer Picard information concerning the complicity of the banks," Madoff wrote. "I was the only person in my firm that dealt with the officers that handled my account. Not Frank (DiPascali, a Madoff lieutenant who has pleaded guilty to fraud charges) nor anyone else. I have little doubt that the information I could provide would clearly demonstrate the vital role the major banks… played, in the carrying out my fraud, including their role in handling the accounts of my major customers. The ball is in his court."
"There's always the suggestion that there's more that he could help us with," said Picard's chief counsel, David Sheehan. "But when we pressed his counsel to give us an example of that before we start getting on an airplane and fly down to see him, we never get anything that's worthwhile. So we have not taken him up on those offers."
Madoff said he has a theory as to why Picard is not giving him credit for his assistance.
"I certainly can understand his reasons for not acknowledging my role. How could he JUSTIFY HIS FEES," Madoff asked.