In an hour-long interview, Mr Andrews was asked about the Victorian Ombudsman's latest report into the state's public sector.
The report, published on Wednesday, did not find that the public service had been improperly stacked with Labor operatives but it had been politicised in "other, equally pervasive ways".
Mr Andrews denied a person's political persuasion influenced whether or not they were employed in the public service.
"Is it such a bolt from the blue that people who are involved in politics might also be involved in public policy?" he told the Socially Democratic podcast.
"(That they) might know and understand and be committed to serving others?"
The former premier said there were people he appointed to senior public service roles who were previously affiliated with the Liberal Party.
"The notion that someone has got political beliefs, or used to work for a politician at some point in time, means that they're forever prohibited from working for all of us - it doesn't make any sense," he said.
It was not fair to blindly attribute "bad motives" to legitimate roles and appointments, Mr Andrews said.
He also criticised the heads of the Victorian Ombudsman and the Independent Broad-based Anti-corruption Commission.
"There's not an accountability officer that doesn't want more money, more power - that's fine, that's their job," Mr Andrews said.
"They have opinions and views, and they're more than entitled to those.
"They're not entitled to pretend that anyone voted for them ... that they've somehow got a mandate that is equal to, let alone superior to, the duly elected government."
The podcast is the first wide-ranging interview Mr Andrews has done since he stepped down as premier in September.
He became Victoria's longest-serving Labor premier earlier this year after close to 10 years in the role.
Mr Andrews said he was proud he "got s*** done" during his tenure.
"We didn't do the popular stuff - we lived our values, our politics," he said.
