President Andry Rajoelina's office offered no details on who was behind the attempt and no signs of violence were immediately visible on the streets on Sunday, although there was a large military presence.
A commander of the elite CAPSAT unit, Colonel Michael Randrianirina, denied any coup had taken place, but the unit claimed to have taken control of all of Madagascar's armed forces and said it had installed a new leader of the military, General Demosthene Pikulas.
"We responded to the people's call," Randrianirina told reporters. Pikulas, who spoke alongside Randrianirina, declined to say if they had asked Rajoelina to resign, but the CAPSAT army unit appeared to be in a position of authority.
Madagascar has been shaken by three weeks of the most significant unrest in years in the nation.
The protests were led by a group calling itself "Gen Z Madagascar," and the United Nations says the demonstrations have left at least 22 people dead and dozens injured. The government has disputed this number.
The whereabouts of the president were not immediately known on Sunday.
His office said he "wishes to inform the nation and the international community that an attempt to seize power illegally and by force" has been "initiated".
"In view of the extreme gravity of this situation," the president's office "strongly condemns this attempt at destabilisation and calls upon all forces of the nation to unite in defense of constitutional order and national sovereignty," it said.
CAPSAT is the same army unit that was pivotal in a 2009 military-backed coup that first brought Rajoelina to power as the head of a transitional government.
A turning point in these protests came on Saturday when soldiers from CAPSAT joined weeks-long anti-government demonstrations and called for Rajoelina and government officials to step down.
Saturday's protests were among the largest since the unrest began on September 25 and Randrianirina said his troops had exchanged fire with security forces who were attempting to quell the protests and that one of his soldiers had been killed.
Speaking to crowds from an armoured vehicle, Randrianirina said that Rajoelina, his new prime minister, the minister of the gendarmerie and the commander of the gendarmerie "must leave power. That's all."
"Do we call this a coup? I don't know yet," Randrianirina had said.
Madagascar, a large island of 31 million people off the east coast of Africa, has had several leaders removed in coups and has a history of political crises since it gained independence from France in 1960.
The 51-year-old Rajoelina first came to prominence as the leader of a transitional government following a 2009 coup that forced then-President Marc Ravalomanana to flee the country and lose power.
Rajoelina was elected president in 2018 and re-elected in 2023 in a vote boycotted by opposition parties.
As the news of Rajoelina's statement broke, the US Embassy in Madagascar advised American citizens to shelter-in-place due to a "highly volatile and unpredictable" situation.
The African Union urged all parties, "both civilian and military, to exercise calm and restraint".
The youth-led protests first erupted last month over electricity and water outages but have snowballed into larger dissatisfaction with the government and the leadership of Rajoelina.
Protesters have brought up a range of issues, including poverty and the cost of living, access to tertiary education, and alleged corruption and embezzlement of public funds by government officials and their families and associates.
with AP
