I've just finished reading The Tao of Physics by Fritjof Capra. I won't undertake a general review of the book, but I can enthusiastically recommend it.
Instead, I want to concentrate on one specific point Capra discusses in the Afterward of his Second Edition. (The book was originally published in 1975, and the second edition was released seven years later.) Capra discusses the ramifications of the empirical results from tests of John Bell's inequality.
As a staunch proponent of a deterministic, local, realist interpretation of quantum mechanics in an earlier day, I had come across Bell's Inequality before. I even wrote about it, which you can verify at the link above. I'm not going to recap that whole post here, so you may want to visit it before continuing. Capra helped me realise that I hadn't really got it, though.
Bell set out a testable relationship that must hold if sub-atomic reality is both local and deterministic. Tests have been conducted, and the results are conclusive: quantum reality must be either non-local, non-deterministic or both.
Einstein believed to his core that reality must be both local and deterministic. I paraphrase his words on each point:
For reality to be local means that no force or action can act with a speed faster than the speed of light. My pushing a button while standing on the sun and immediately changing the channel on my TV would violate this, since it takes light seven minutes to traverse that distance.
For reality to be deterministic means that every effect has a cause. Taken to its logical conclusion, the notion is embodied in the clockwork universe. If one knew all the initial conditions at the 'start of time' and all of the 'laws' that the universe followed as it evolved from one moment to the next, then one could know all that would happen for all time.
We use probabilities in everyday life (for instance, at the roulette wheel) because either our understanding of all the (local) forces at work or the exactness with which we can measure initial conditions is insufficient to calculate outcomes exactly. Quantum mechanics raises the possibility that even with perfect understanding and measurement, we still could not accurately predict concrete sub-atomic events.
I was particularly exercised by the issue of determinism, so when I read about outcomes from tests of Bell's Inequality, I focused on that element of the interpretation. I admitted, sadly, that not every effect has a cause. I need not have done so, had I paused to think sufficiently about the second condition. It could be that tests failed the inequality because the causes were non-local. I could, in theory, have held on to determinism by accepting spooky action at a distance.
But in a way, admitting non-locality undermines predictability just as much as inherent non-determinism does, because an observer cannot know and therefore account for forces at great remoteness in the universe that are impacting what he sees. Even if he had appropriate sensors throughout the cosmos, they could not transmit their data to him any faster than the speed of light. Meanwhile, the spooky action at a distance will have had immediate effect.
As it happens, today, I believe that reality is not determined, local or objective. Most interpretations of quantum mechanics agree, but my angle is primarily from the philosophical viewpoint rather than the scientific. A worldview consistent with eastern wisdom traditions sees reality as an undivided whole, so the confirmation of non-locality is no surprise.
In a way, you can think of it as rotating causality 90 degrees in space-time. Instead of explaining a current event by appealing to causes that preceded it, you sometimes have to explain it in terms of state of everything else right now. Sometimes, the best that we can do is say that something is the way it is because everything is the way it is. Any one 'thing' is like a puzzle piece, which, in order to fit into the overall puzzle, can have one and only one shape - the shape of the 'hole' left when every other piece is in place. There are limits to reductionism, and we have touched them. At least some truths are irreducible.
Of course, this philosophical sleight of hand doesn't help with predicting the future. We just have to accept that, as we push the boundaries of our understanding of the universe, sometimes the best we can do is approximate or confine our answers to a range rather than a point. There are limits to our knowledge and the control we can exercise with it.
I'm curious. I like looking beneath and behind the obvious, also looking for what is between me and the obvious, obscuring or distorting my view.