OK... Let me see if I can clear this up with you. According to what I have learned from a Christian pastor, when the lost die they are judged and they go to the great white thrones of death. They are not there to stay. The lost could be told something like this, "Depart from us, we never knew you."
Do you follow me so far?
Once they are cast away, where do they go? They go somewhere where they are potentially ganged up on because they are not protected. I believe God essentially is telling us that some are not suited to live in God's realm then, the lost are cast away from there.