Cuomo, 67, who had been seeking a political comeback four years after resigning amid sexual harassment allegations, told supporters he had called Zohran Mamdani to congratulate him.
"Tonight is his night," Cuomo said on Tuesday while conceding the race to Mamdani.
Mamdani, who entered the campaign as a virtual unknown, was ahead of Cuomo 43.5 per cent to 36.4 per cent with nearly 95 per cent of ballots counted.
Nine other Democratic candidates trailed far behind.
The outcome will not be final until next week due to New York's ranked-choice system that allows citizens to pick up to five candidates in order of preference.
But Mamdani's lead in Tuesday's preliminary results appeared too large for Cuomo to overcome.
Mamdani told supporters: "I will be the mayor for every New Yorker, whether you voted for me, for Governor Cuomo, or felt too disillusioned by a long-broken political system to vote at all.
"I will work to be a mayor you will be proud to call your own."
The race was seen as an indicator of what direction Democrats believe the party should take five months into Republican President Donald Trump's tumultuous second term.
Their differences were clear: Cuomo, a moderate backed by the establishment who served a decade as governor, or Mamdani, a progressive newcomer who promised a break with the past.
Mamdani will likely be the favourite in November's general election in a city where Democrats dominate.
The current mayor, Democrat Eric Adams, will appear on the November ballot as an independent, but corruption scandals and his perceived ties to Trump have weakened his standing.
The Republican candidate is Curtis Sliwa, a radio host best known as the founder of the Guardian Angels anti-crime patrol who lost to Adams in 2021.
Cuomo has denied the harassment accusations, which he characterises as ill-conceived attempts to be affectionate or humorous.
Uganda-born and of Indian descent, Mamdani who would be the city's first Muslim mayor and has a history of pro-Palestinian activism.
He was elected to the state assembly in New York's Queens borough and has garnered the support of US senator Bernie Sanders and House representative Alexandria Ocasio-Cortez, two prominent progressives.
Cuomo, who emerged as a vocal critic of Trump during his first term as president, won the endorsements of former president Bill Clinton and former New York mayor Michael Bloomberg.
