The Labour grandee was sacked on Thursday after emails were published showing he had sent supportive messages to Jeffrey Epstein, even as the paedophile faced jail for sex offences.
Giuffre, who died in Western Australia in April aged 41, was one of the most outspoken accusers of convicted sex offenders Epstein and his former girlfriend Ghislaine Maxwell.
Her brother, Sky Roberts, and sister-in-law, Amanda Roberts, gave their first broadcast interview in the UK on the BBC's Sunday With Laura Kuenssberg.
Asked if Lord Mandelson should have been given the role in February, Mr Roberts said: "Absolutely not. He should not have been given the position in the first place."
"Why does it take us to have to pull out the skeletons for people to be held accountable?" Roberts said.
"Our governments have allowed these people to hold their status and their title without shame, and so part of me is like, we should have done something sooner.
"He should have never been given that title."
Giuffre also sued the Duke of York for allegedly sexually assaulting her when she was 17 after she was trafficked by Epstein, which eventually led to Andrew paying millions of pounds to settle the case.
She alleged they trafficked her to the duke, with the duke denying the claim and saying they never met.
Andrew stepped down from public life after the furore over his friendship with the paedophile financier.
Roberts said the royal family had not reached out to their family in any way following the settlement, and that she believed more revelations would "come to light".
A memoir of Giuffre's life, titled Nobody's Girl, is scheduled for release in October.
1800 RESPECT (1800 737 732)
National Sexual Abuse and Redress Support Service 1800 211 028
12°C