The school I went to tried to do that to a mate of mine, only rather than ask him to leave they suggested he do a 'non-ATAR VCE', which meant he'd stick around and still pay them fees but not get an ATAR score when he finished (apparently it would 'take the pressure off him').The high end public schools also use the trick of “moving on” the lumps at the end of year 10, though I’m sure the tactics are a lot more prevalent in the elite private schools.
In saying that, if your kid is academically inept, there’s more incentive to move them out of those schools before year 12 if they have no aspirations or need to get a decent ATAR, so the parents can save on forking out the massive $$$ in fees.
After having handed over all of that money over the years right up until year 12, he and his parents promptly told them to get ****ed, completed his year 12 like everyone else and happily dragged the school's average down with him