While he did not mention Ivanka Trump and her husband Jared Kushner by name, Kelly, 48, said in an interview with Bloomberg Television’s The David Rubenstein Show that the two married staffers were “an influence” that had to be handled during his time in the Trump administration.
“They were an influence that has to be dealt with,” Kelly replied after being asked if it was complicated to have the president’s family working at the White House during his tenure as chief of staff.
Kelly immediately qualified his statement, adding that he was not referring to First Lady Melania Trump with his remarks.
“By no means do I mean Mrs. Trump,” he said. “The first lady’s a wonderful person.”
The White House did not immediately respond to PEOPLE’s request for comment.
Apparently, at one point while Kelly was chief of staff, Trump reportedly told him to fire Kushner and Ivanka.
According to Vicky Ward’s book Kushner Inc., which was based on 220 interviews, Trump told his chief of staff, “Get rid of my kids; get them back to New York.”
Trump’s command matches a report by the Washington Post in March 2018.
“ has mused to Kelly that he thinks Ivanka and her husband should perhaps return to New York, where they would be protected from the blood sport of Washington and less of a target for negative media attention, White House officials said,” the Washington Post‘s Ashley Parker and Philip Rucker wrote.
However, in response to Ward’s book, the White House denied her written account of Trump’s reported conversation with Kelly.
“It’s sad, but not surprising, the media would spend time promoting a book based on shady anonymous sources and false information instead of all the incredible work Jared and Ivanka are doing for the country,” Press Secretary Sarah Sanders said in a statement to PEOPLE.
RELATED: Jon Voight Will Be a President Trump Appointee for the Next 6 Years … Here’s What That Really Means
Peter Mirijanian, who spoke on behalf of Kushner attorney Abbe Lowell, also denounced Ward’s account in a statement to the New York Times, which initially preview Ward’s manuscript in March.
“Every point that Ms. Ward mentioned in what she called her ‘fact checking’ stage was entirely false,” he said. “It seems she has written a book of fiction rather than any serious attempt to get the facts. Correcting everything wrong would take too long and be pointless.”
Read more: people.com