Rajoelina had refused to step down despite escalating Gen Z demonstrations demanding his resignation and widespread defections in the army.
"We have taken the power," Colonel Michael Randrianirina declared on state radio and said that the military was dissolving all institutions except the lower house of parliament or National Assembly.
Randrianirina later told reporters a committee led by the military would rule the country for a period of up to two years alongside a transitional government before organising new elections.
"The following institutions are suspended: the Senate, the High Constitutional Court, the Independent National Electoral Commission, the High Court of Justice, and the High Council for the Defense of Human Rights and the Rule of Law," a statement from the country's military leaders said.
In a day of turmoil for the country off southern Africa's coast, the 51-year-old leader, whose whereabouts are unknown, earlier sought to dissolve the lower house by decree.
But MPs went ahead with a vote to impeach him, leaving the country in a constitutional deadlock which the military seized upon to declare they were taking charge.
Rajoelina, who himself came to power in a coup in 2009, condemned the power grab by the military in a statement.
Randrianirina, a commander in the elite CAPSAT army unit that played a key role in Rajoelina's 2009 coup, broke ranks with him last week.
In a defiant address on Monday night, Rajoelina said that he had been forced to move to a safe place because of threats to his life.
An opposition official, a military source and a foreign diplomat told Reuters he had fled the country on Sunday aboard a French military plane.
His isolation increased further on Tuesday when even MPs from his ruling coalition, which holds a parliamentary majority, voted to impeach him on charges of engaging in activities deemed incompatible with presidential duties.
Rajoelina had repeatedly warned in recent days that an attempted coup was under way in the country.
Demonstrations first erupted in the country on September 25 over water and power shortages and quickly escalated into an uprising over broader grievances, including corruption, bad governance and a lack of basic services.
The anger mirrored recent protests against ruling elites elsewhere, including Nepal and Morocco.
Earlier on Tuesday, at Antananarivo's 13 May Square, along the main drag lined with palm trees and French colonial buildings, thousands of protesters danced, marched, sang and waved banners denouncing Rajoelina as a French stooge because of his dual citizenship and support from Madagascar's former coloniser.
Many were waving Malagasy flags and the signature Gen Z protest banner of a skull and crossbones from the Japanese One Piece anime series.
At one point, Randrianirina took the stage and asked: "Are you ready to accept a military takeover?," drawing cheers of approval from the crowd.
Later, as news of the military takeover filtered through to protesters, many were jubilant.
"We're so happy Andry Rajoelina is finally gone ... We will start again," high-school student Fih Nomensanahary said, with four of her friends cheering alongside her.
Others were more cautious.
"They need to hand over to a civilian administration quickly and have an election," said Rezafy Lova, a 68-year-old IT consultant.
CAPSAT had joined the protesters over the weekend, saying it would refuse to fire on them.
It went on to take charge of the military and appointed a new army chief, prompting Rajoelina to warn on Sunday of an illegal attempt to seize power.
Since then, the paramilitary gendarmerie and the police have also broken ranks with Rajoelina.
Madagascar, where the average age is less than 20, has a population of about 30 million, three-quarters of whom live in poverty.
Between its independence in 1960 and 2020, GDP per capita plunged 45 per cent, according to the World Bank.
