A 3-2 defeat for the Aussies leaves their hopes to reaching the knockout stage hanging by a thread ahead of their final Group B game next week against Canada.
Australia went ahead in first-half injury time through Emily van Egmond but Uchenna Kanu soon equalised and Osinachi Ohale put Nigeria in front midway through the second half.
Asisat Oshoala then made it 3-1 and Nigeria held on for a famous win despite a late goal from Alanna Kennedy.
Nigeria move top of Group B, level with Canada on four points.
Australia are a point behind before the last round of Group B matches on Monday, when they face Canada in Melbourne needing a win to guarantee a place in the last-16.
Nigeria play already-eliminated Republic of Ireland in Brisbane.
The Aussies are still sweating on the fitness of star player Sam Kerr, who has missed their first two games through injury.