Mr Latham is accused by his former partner Nathalie Matthews of a "sustained pattern" of abuse and degradation.
The NSW state MP and one-time prime ministerial hopeful strongly denies the untested claims made in a civil court apprehended violence order application by Ms Matthews, saying he has "broken no laws".
The Labor caucus on Monday agreed that Mr Latham's official portrait in the federal party room would remain, but with a caption providing context.
Weighing in for the first time since the allegations emerged, the prime minister said he never wanted Mr Latham to be his party's leader.
"Mark Latham has views which I find repulsive across a range of areas," Mr Albanese told ABC's 7.30.
"He's someone who I regret being ever being elected leader of the Labor Party."
Mr Albanese said his feelings were not in "retrospect" as he had been doing the numbers for Kim Beazley during the 2003 leadership challenge which Mr Latham narrowly won.
"History has proven that judgment to be correct. Mark Latham since ... has gone further and further and further away from any values that represent mainstream Australia," he said.
The words underneath the portrait will read: "In 2017 Mark Latham was expelled from the Australian Labor Party and banned for life. His actions do not accord with Labor values and fail to meet the standards we expect and demand."
Mr Latham sits as an independent in the NSW upper house and faces calls to resign over sexually explicit messages allegedly sent to his former partner while sitting in the chamber of parliament.
The decision to add context to Mr Latham's portrait was the right one, Finance Minister Katy Gallagher said.
"It's finding the balance, you can't erase history, we can't pretend he wasn't our leader, he was and so there he sits in the leaders' timeline," she told reporters in Canberra."But it's a recognition ... his behaviour and attitudes don't reflect the modern Australian Labor Party."
Senator Gallagher said the wording allowed people to feel something had been done.
"It will exist there forever on our leaders wall," she said.
"It's a pretty strong statement."
Mr Latham lost his bid for the nation's top job at the 2004 federal election to former Liberal prime minister John Howard.
The campaign was marked by his aggressive handshake with Mr Howard outside the ABC's radio studios on election-eve.
The infamous episode was largely blamed for his election defeat and delivered the Howard government a fourth term.
In 2024, the Federal Court ordered Mr Latham to pay independent NSW politician Alex Greenwich $140,000 in damages over a homophobic social media post.
1800 RESPECT (1800 737 732)
Lifeline 13 11 14
Men's Referral Service 1300 766 491
