Yes, there are non-deterministic parts in physics. For example atomic decay. While we can measure and work with half-life times for large amounts of radioactive atoms, the decay of a single, individual atom is unpredictable. So in a way, you can get your desired dose of vagueness by controlling how many atoms you monitor. The less, the more.
Or another example from the same field: There are atoms for which we believe they are stable, although they theoretically could decay. But we never observed it. So maybe they are in fact stable, or maybe they decay just slower than we have time. Or only when we don’t look. Examples would be Helium-4 or Lead-208.
I also like the idea, inspired by Douglas Adams, that the universe itself could be a weird and random fluctuation, which just happens to behave as if it was a predictable, rationally conceivable thing. That actually, it’s all a random chain of junk events, and we’re fooled into spottings some patterns. This apparence could last forever or vanish the very next moment, who knows. Maybe it’s all just correlation and there is zero causation. As far as I know, we’ll never be able to tell. So fundamentally, all of it is a vague guess, supported by mountains of lucky evidence.