We are all immigrants in Australia, unless you are Aboriginal. Whether you came here as a convict or settler, or as a refugee (such as my own family, from Germany and France), or as a skilled migrant, we are all immigrants. We have no more 'right' to be here, than anybody else.
History shows that whenever times get hard, the ugly face of intolerance and bigotry show up. As Peggyaus says, this is often on both sides. Media beat ups, and orchestrated distractions, keep this whole stupid argument going, fueling the hatred and suspicion. The Aboriginal people in Australia are the only ones that can claim a 'cultural heritage', for anyone else over here, our 'culture' is diversity, for 200 and a bit years we have been a mish-mash of different cultures and peoples.
This is an old song I remember from my childhood, funny how nothing changes.