2020-05-06

Predictions are overrated?

IMG_20200506_100300 Sabine Hossenfelder has a blog post (via Not Even Wrong) asserting that Predictions are overrated. Obviously, in PW's words, the naive view that you can evaluate a physical theory simply by the criterion “Does it make predictions?” is wrong; but SH goes too far in dissing predictions.

Happily, after all the physics, which they definitely know more about than me, SH comes on to climate models:
Another example where this misunderstanding matters are climate models. Climate models have correctly predicted many observed trends, from surface temperature increase, to stratospheric cooling, to sea ice melting. That’s an argument commonly used against climate change deniers. But the deniers then go and dig up some papers that made wrong predictions. This, so their claim, demonstrates that really anything is possible and you can’t trust predictions.
This is naive to the point of being wrong. Most importantly, most climate modelling - well, especially the IPCC - has been careful to talk of projections rather than predictions. The outside world hasn't been great at picking up that nuance, but it is there, and needs to be considered if you're attempting some scientific evaluation rather than a political one. That we don't know the value of climate sensitivity is well up front, and clearly century-scale predictions of climate evolution aren't possible without that. Secondly, some of the predictions she mentions - sea ice melting is the most obvious - are ones where the models have done a fairly poor job, other than getting the sign right. And thirdly, there's a variety of models predicting different things, and they can't all be right.

How you should evaluate the credibility of the climate modelling community / scientists / effort, based on past "predictions"? Clearly, picking any one "prediction" at simply verifying that is wrong. You need to look at more of an amalgam, like the IPCC. As to the kinda question "should we trust them now, based on what they said in the past?" you need to look at how the predictions were presented. If you could find multiple frequent cases where people confidently published clear predictions which were subsequently proved wrong, then you would indeed mark them down. Since that isn't actually the case, you don't.

And then there's the point, which SH notes that we have moved on to arguing about the integrity of scientists and the policies of their journals instead about science. If you're talking about denialists, then yes, you're talking about "integrity" and sociology of science, not about actual science. What if you're actually interested in evaluating the science? Then I still think the ability of the models to make predictions matters. In this case it's hard to say quite what we mean by "the science" - all the subcomponents like radiative transfer theory are effectively "unit tested" to borrow from software engineering, but those aren't the bits we're evaluating; in terms of GW, "the science" means that integrating them all together within a GCM (a) works and (b) captures enough of the physical world to say something useful. Without something in the way of prediction, I don't see how that's possible. Where generally you allow "prediction" to include predicting the past; i.e. temperature evolution over the 20th century.

SH proposes instead What, then, is the scientific answer for the climate change deniers? It’s that climate models explain loads of data with few assumptions. Which is nice, but never convincing, unless you believe it anyway. It isn't convincing because it isn't clearly true from the outside: you can't tell that they have "few" assumptions (hence, it is a useless answer for denialists themselves, and not much use when in front of an audience of the general public). Worse, the statement isn't even meaningful; GCMs have lots of "assumptions" in them, measuring "few" or "lots" in any meaningful way would be a difficult task in itself. This is why predictions are good: they don't require looking inside the black box. Perhaps it is just her rhetorical question that's wrong: if she'd asked, What, then, is the scientific answer for the climate change scientists? it would make some sense, but still be hard to evaluate.

Just to be clear, I don't think there's an absolute answer. Trying to evaluate a scientific theory and coming up with an opinion as to whether it is likely true or not isn't a science. There's a question of how much weight you give to predictions versus other factors. But making good predictions is generally so hard, anything that can predict correctly gets a lot of weight.

7 comments:

Tom said...

Actually, it's the hidden assumptions that bother some. I spent a day on a thread with ATTP trying to show him that the RCPs assumed sensitivity of 3C. I showed him the link, quoted the language--but I don't think I convinced him. He's certainly never mentioned it since.

Old_salt said...

Tom,
You don't understand climate sensitivity. It is not input into the models but gets determined by the interactions of the different parts of how the model describes the climate system. It wsa in fact first used as a metric to compare models--double CO2 and see what the equilibrium temperature became.

So, there is no assumed sensitivity.

...and Then There's Physics said...

Actually, it's the hidden assumptions that bother some. I spent a day on a thread with ATTP trying to show him that the RCPs assumed sensitivity of 3C.

IIRC, what you were referring to was that some of the models that are used to associate the RCPs with emissions assume a climate sensitivity of 3C. I may remember incorrectly, of course, but what you clearly did not show was that the RCPs assume a sensitivity of 3C, since RCPs are simply Representative Concentration Pathways.

William M. Connolley said...

Os: Tom isn't talking about the (common denialist meme) of CS built into the GCMs (of course). But in some connection with RCPs. With which I'm not familiar TBH, though something in the RCP process assuming some CS wouldn't be strange. I don't think there are any hidden assumptions; though there are lots of not-well-known assumptions, even in GCMs, since they are Quite Large.

...and Then There's Physics said...

I think I've found the comment Tom made on my blog, but it still doesn't make much sense to me. It's possible he's talking about MAGICC, which is reduced complexity model and is associated with the development of the RCPs and might have an assumed ECS of 3C (although I'm not sure). As you say, though, these assumptions aren't exactly hidden and I would still argue that statement "the RCPs assumed sensitivity of 3C" doesn't make sense.

...and Then There's Physics said...

Actually, I found another on of Tom's comments which refers to this paper. The key quote is
As concentrations are somewhat dependent on the future climate itself (due to climate feedbacks in the carbon and other gas cycles), we emulate median response characteristics of models assessed in the IPCC Fourth Assessment Report using the reduced-complexity carbon cycle climate model MAGICC6.
which does indeed seem to be referring how you associate emissions with concentrations.

Dikran Marsupial said...

"SH proposes instead What, then, is the scientific answer for the climate change deniers?"

There isn't one, there is no such thing a proof beyond unreasonable doubt and they are not listening.

The scientific answers are for the people that are listening to the deniers, but don't understand the science well enough (yet) to know the answers to their canards.