Exit polls by major Israeli TV stations showed neither Netanyahu's political bloc nor that of his main rival, former military chief Gadi Eisenkot, secured the 61 seats needed for a parliamentary majority.
Despite projections pointing to a setback for Netanyahu's coalition, the prime minister said his right-wing Likud party had emerged as "the largest party in Israel."
Netanyahu, 77, accompanied by his wife Sara, Netanyahu appeared upbeat, clapping his hands and thanking his supporters.
He also slammed the country's Arab-led parties, which exit polls suggested expanded their strength in parliament.
"We must not form a government that relies on Arab, anti-Zionist parties," Netanyahu said.
Turnout was 77.7 per cent, Israel's Central Elections Committee said, the highest in more than two decades and around seven percentage points higher than in 2022.
Two of Israel's three major television networks gave Eisenkot's bloc a narrow edge over Netanyahu's Likud-led camp in updated exit polls, while Channel 13 put the two blocs level at 51 seats each.
None of the projections gave either camp the 61 seats needed for a majority in the 120-seat Knesset. Far-right religious parties led by Itamar Ben-Gvir and Bezalel Smotrich were projected to take a combined 13 seats within Netanyahu's bloc.
Final results could take days, and Israel's fragmented political system leaves smaller parties potential kingmakers in coalition negotiations that have run to weeks or months after past elections.
Eisenkot said the public had voted against Netanyahu remaining prime minister.
"We have made history! The public has voted for change," he told supporters.
"The Israeli public has voted against Netanyahu continuing to serve as prime minister of Israel.
"Israel is on the path to a new era - an era of hope, mutual responsibility and solidarity."
Tuesday's vote was the first election since the Hamas-led attack of October 7, 2023, and the ensuing wars in Gaza, Lebanon and Iran.
Yohanan Plesner, president of the Israel Democracy Institute, called the exit polls a major defeat for Netanyahu's coalition, while cautioning that Eisenkot had not emerged as a clear winner and much could change as votes were counted.
Netanyahu, 77, has served as prime minister for nearly 19 of the past 30 years and has repeatedly survived political crises. He entered the election facing longstanding corruption charges, which he denies, and criticism over intelligence failures that allowed Hamas-led militants to kill 1200 people and take 251 hostages.
Netanyahu's opponents have managed to wrest him from the premiership only once since 2009, and that took four inconclusive elections in two years.
He was back 18 months later.
For many of Netanyahu's opponents, the election turns on accountability for the security failures of October 7.
Opposition groups have promised to hold an inquiry, which Netanyahu's government has resisted.
Eisenkot walked to the polls with Gadi Mozes, one of 251 people kidnapped by Hamas and taken to the Gaza Strip during the attack that killed 1200 people.
He says Israel's response - a war which health authorities say killed more than 74,000 Palestinians - was a failure because Hamas remains in charge.
He repeated his commitment to dismantle or destroy Hamas' military and governing power in the Gaza Strip within 18 months of forming a government.
Itamar Ben-Gvir, leader of an ultranationalist party in the governing coalition, cast his vote in the Kiryat Arba settlement in the occupied West Bank.
He predicted Netanyahu would return as prime minister.
Ahmed Tibi, a candidate from an Arab party, said a record turnout from Israel's Arab minority would "pave the way for an alternative government and change the grave reality".
with DPA and AP