The former Test opener made the bombshell allegations eight years on from the 2018 incident at Newlands.
An official investigation determined only David Warner, Steve Smith and Cameron Bancroft knew of the idea to use sandpaper in the third Test.
But Khawaja, who batted at No.3 in that match, rubbished those findings.
"It was all of us," the retired left-hander told Nine Newspapers as he prepared to release his memoir Sabr.
"Not the batters, not the bowlers, not the staff. All of us.
"I don't want to be the one who comes out and talks about the truth.
"The issue is that I genuinely thought after eight years the truth would have been out by now."
Bancroft received a nine-month suspension, while Warner and Smith were banned for a year.
Other players including bowlers Pat Cummins, Mitchell Starc, Josh Hazlewood and Nathan Lyon distanced themselves from the incident, insisting they had no knowledge "of a foreign substance" being taken on the field.
In excerpts of his book, Khawaja recalled open, lighthearted discussions were held in the dressing room at St George's Park in Port Elizabeth, now Gqeberha, about how to rough up the ball.
The retired opener said Smith walked away when sandpaper was found to be effective, suggesting he didn't want to know about it.
At a break in play, Khawaja said Bancroft then volunteered to use sandpaper in South Africa's first innings in Gqeberha and the ball swung lavishly during that period.
Despite Khawaja's admissions, Cricket Australia has defended the findings of its swift probe.
"There has been a great deal of speculation in the eight years since the 2018 Cape Town Test, including extensive media coverage throughout last month's series in South Africa," a CA spokesperson said in a statement.
"During this time there has been the ongoing opportunity to present any relevant information and this remains the case.
"All relevant information presented to CA to date has been properly considered and we remain comfortable with the findings of our integrity process."
Khawaja said he spoke passionately in the change rooms after the game for the team to take collective responsibility, rather than scapegoating Warner, Smith and Bancroft.
"Obviously, it didn't work that way and those three guys got crucified," he said.
Darren Lehmann, who stepped down as Australia coach at the end of the ugly series, stopped short of admitting specific knowledge of the use of sandpaper before Bancroft was caught in the act by TV cameras.
"I don't think everybody would have known," Lehmann said on ABC Grandstand on Saturday.
"I can't speak for them (the players) at all. I only know what I know, and I know that ball-tampering was a big part of the game of cricket.
"Now it's obsolete, but that doesn't make it right either and we paid a heavy price for that."
Lehmann said he had no memory of Kahwaja pleading his case for collective responsibility, but didn't outright dismiss the claim, noting there were two change rooms and he was in meetings.
The former coach said the story dredged up "bad memories" and took a thinly veiled swipe at Khawaja.
"Eight years after the fact and trying to sell a book's a bit different, that's Usman's choice," Lehmann said.
Khawaja gave Cummins a "heads up" about his upcoming autobiography in the lead up to Australia's recent tour of South Africa.
The Australian Test captain was unconcerned about the possibility of any sandpapergate revelations reopening old wounds.
"It's obviously a long time ago," Cummins said last month.
"(I've) answered every question a million times so it was dealt with then ... all I can say is, as a group, we've moved on."
8°C