Is it wrong, now, for you and your culture to invade someone else and forcibly remove them from their land?
If no, then you don't have modern morality and you're likely making an argument like "we conquer people all the time, might makes right" etc.
If yes, then you have to apply that to benefits you and your society received and are still benefiting from. And that means making redress for past wrongs you personally didn't commit. Otherwise the logic follows that the sin or crime only follows to the direct perpetrator and transferring that benefit to another absolves that person, especially when transferal wasn't voluntary (inherited).
This doesn't mean that descendants of colonizers must vacate all land. It means the society needs to have a frank and honest discussion about what redress means. This is at the heart of similar movements like slavery reparations. We can't undo the wrongs of the past but we can stop making them and try to make it right to the descendants of those that were harmed.
Now this obviously can't be taken back all the way into prehistory. As everyone acknowledged: humanity is a brutal species. But we can make redress for the most recent and often still ongoing wrongs. We should also attempt to address past wrongs but to the level appropriate to the passage of time, existence of descendants, etc.
We can see this happening now with the return of art that colonizers and other thieves took from other countries. We can't undo the theft, but returning to the originating culture is a feasible solution. When talking about land stolen en masse a few hundreds of years ago, and the terrible atrocities committed to a still-existing culture to enforce that theft, there's a variety of solutions, none of which are total return of all land. In all things we can't unduly punish those existing now that didn't have a say while also trying to do right to those still burdened by past atrocities.