In some sense it is violence, but in another sense violence against a baby doll is fine while violence against a live baby is not. One of the key differences is the baby is sentient, the doll is not. The doll can not experience the violence, so it’s morally irrelevant. The baby can experience the violence, so it is morally unacceptable. To the best of my knowledge plants can’t feel pain, so violence to them is morally irrelevant.
But I still accept that harm is unavoidable (at least in our time), but our response should be to minimize harm, not throw up our hands and give up and perpetuate the injustice