Former information minister Fawad Chaudhry's resignation is the latest - and highest profile - in a string of departures from Khan's Pakistan Tehreek-e-Insaf (PTI) party, which the civilian government on Wednesday threatened to ban.
"I have decided to take a break from politics, therefore, I have resigned from party position and parting ways from Imran Khan," Chaudhry said in a post on Twitter.
Chaudhry spent days in detention after violent protests swept the country this month after the detention of Khan on corruption charges.
He condemned the protests by Khan's supporters, who attacked military installations including army headquarters as well as government buildings.
Khan says the corruption allegations were fabricated and that his associates are being forced out under duress from the government and the military in a manoeuvre to dismantle his party before elections scheduled later this year.
He has been embroiled in a tussle with the military since he was removed from power last year in a parliamentary vote which he says was orchestrated by the country's top generals.
The military denies this.
Khan is Pakistan's most popular leader according to local polls while the military is its most powerful institution, having ruled directly or overseen governments throughout Pakistan's 75-year history.
The face-off has raised new fears about the stability of the nuclear-armed country of 220 million people as it struggles with its worst economic crisis in decades.
Chaudhry is the second former federal minister to leave Khan.
On Tuesday, former human rights minister Shireen Mazari announced she was leaving politics, citing health concerns, after spending 12 days in detention.
Most of the top leaders of the PTI have been taken into custody.
A number of former parliamentarians and mid-tier leaders have quit the party or politics entirely over the last few days.
Defence Minister Khawaja Asif told reporters on Wednesday that the government is considering banning the PTI for attacking the "very basis of state" and this could not be tolerated.
A ban would be likely to further enrage Khan's supporters and exacerbate the confrontation with the military establishment.
PTI party lawyer Ali Zafar said any such step would be challenged in court.
He said an entire party cannot be blamed for acts committed by individuals.
Khan, 70, became prime minister in 2018 with the tacit support of the military, though both sides denied it at the time.
But he later fell out with the generals after being seen as having tried to interfere in key promotions in the security sphere.
After being removed from power last year, Khan has been campaigning for a snap general election and has rallied supporters across the country.
But the prime minister who replaced him, Shahbaz Sharif, has rejected calls for a vote ahead of the due date late this year.
Khan has said the corruption charges were made up to banish him from politics.
He was detained on May 9 sparking the protests by his supporters and their assaults on the military facilities.
He was later freed on bail.
18°C