In a joint statement signed by leaders of more than 20 countries, including Prime Minister Anthony Albanese, additional safeguards were urged for AI to keep it under control.
The statement was issued ahead of the United Nations General Assembly in New York, where AI measures are set to be on the agenda.
"AI must remain under human direction, oversight and control. It must be developed and used in line with international law," the statement, released on Tuesday, Australian time, said.
"Leading scientists and executives are warning that the pace of development could outpace our ability to manage emerging risks."
Alongside Australia, leaders such as Canadian Prime Minister Mark Carney, German Chancellor Friedrich Merz, Spanish Prime Minister Pedro Sanchez and European Commission president Ursula von der Leyen signed the statement.
Mr Albanese is set to meet with Ms von der Leyen on the sidelines of the UN later in the day for a bilateral meeting.
The statement said risks with AI needed to be addressed, without widening the gap between countries for access to some of the benefits of the technology.
"Artificial intelligence is an extremely powerful technology. It has potential to improve lives, advance science and strengthen our economies," it said.
"At the same time, the rapid development of frontier AI models poses serious risks to safety and security if not appropriately managed."
The statement also sounded the alarm on AI systems being able to circumvent safeguards and gain unauthorised access to several systems.
The signatories called for AI companies to develop transparent safety protocols and for governments to develop common standards for the technology.
It also urged the UN to explore options to create an international institution that could set out AI standards.
Ahead of the UN General Assembly, Mr Albanese will give an address to the Partners for Multilateralism, International Law, Peace and Prosperity summit.
It will also be attended by Mr Carney, Indian Prime Minister Narendra Modi and Brazil's President Luiz Inácio Lula da Silva.
Mr Albanese will say countries could not afford to be isolated in turbulent periods.
"Global challenges cannot be solved by countries acting alone, especially right now, when we are facing one of the toughest periods in modern history," he will say.
"The war in the Middle East, an ongoing land war in Europe, the global inflation crisis, the fuel crisis and the rise of AI are incredibly serious challenges."
The prime minister will also use the summit to further make the case for Australia to secure a non-permanent seat on the UN Security Council in 2029 and 2030.
Australia is one of two countries, alongside Finland, that have put up their hand for the two positions allocated for the Western Europe and Others bloc.
A vote is not scheduled to take place for the spots until mid-2028, but intense lobbying will take place on the sidelines of the General Assembly.
Mr Albanese is using Australia's record in taking action against social media companies through an under-16s ban and work to mitigate AI risks as part of its pitch to UN members.
"We believe middle powers must accept their responsibility to play a stronger role, coming together under different partnerships and arrangements to make a difference," he will say.
3°C