The number of vulnerable children in "substandard" residential care is rising in Australia's worst-hit state, causing a $500 million budget blowout.