Stability in a control run of HadCM3

One of the things I do is port [[HadCM3]] to new platforms (although I shouldn't over emphasise my role in that: much of the hard work of portabilising it was done at the Hadley Centre; nonetheless new platforms throw up new problems). HadCM3 was written for a Cray T3E; its known to be stable when run without forcing for thousands of years on that platform. There is a portable version of the model, which requires a little bit of effort to make it run on new platforms. The first thing to do is make it compile; the second to make it run through the first timestep; the third through the first meaning period; and then hopefully all that remains is to check that it is stable.

Which brings in this picture. Black is a 200 year control run, with the g95 compiler on a 4-processor Opteron system (using 3 procs for most of the time). Blue is a rather older run on an Athlon system under the antique fujitsu/lahey compiler. Red is an in between run on Opteron with the Portland Group compiler (pgi). All are seasonal data, differenced from 100 year means of an "official" control run. What you'll notice is that the red run has a distinct climate drift, which is enough to make it unusable. Blue looks OK; black has been run out long enough to be sure its OK. The grey shaded bit is some kind of 95% confidence limit based on the variability of the 100 year "official" run.

Quite why the Opteron/pgi runs drifts I don't know. Its 99.999% the same code as the other runs (differing only in whatever it took to make the compiler accept it). Most likely there is some compiler bug in there; but I will probably never know.

By eye, the 200 year run has no drift. By line fitting, the results are:

0- 50: [ 0.00168505, 0.00451390]
0-100: [ 0.00052441, 0.00159977]
0-150: [-0.00050959, 0.00009494]
0-200: [-0.00060853,-0.00015253]

where I've shown the (95%) confidence intervals for a line fit over the first 50, 100, 150 and 200 years. Which shows up the internal variability quite nicely. If I'd just taken the first 50 years I might have believed in a drift of 0.3 oC/century, which is small but not perhaps totally negligible. By 100 years the "drift" has a central value of 0.1 oC/Century which would be negligible. Out to 150 years there is no statistical trend. Out to 200, a trivial cooling. Note, BTW, that all these sig estimates are rather thrown-together and should be a bit wider to take proper account of autocorrelation.


Anonymous said...

So that explains global warming then. It's not carbon dioxide after all, it's just that the Universe is being run on an Opteron with the Portland Group compiler. GW will stop when God switches to g95.

(How long before this argument appears on skeptic websites, I wonder?)

William M. Connolley said...

Not long I imagine... I did wonder if I should write this post at all. But the information wants to be free.

I like the idea of the universe having the wrong compiler though :-). There was a good piece sometime back about changes to the world as NWP models increased their resolution... mountaineers being thrown up in the air as peaks suddenly appeared... but I don't know where it appeared. Must have been years ago now.

Wolfgang said...


what is your experience with the g95 when it comes to performance and optimization (e.g. compared with ifort from Intel on x86) ?

William M. Connolley said...

g95 is definitely slower, at least in some ways. At the moment its the only compiler I have working on this system (sort of) and I haven't done any systematic tests. My impression is that its a factor of 2 slower than pgi for the ocean code, but may be comparable for the atmosphere. I don't know about ifort: it runs very fast but crashes in the first model timestep!

I did some toy tests once (calculating sin's 10^6 times and such) which indicated to me that g95 was comparable at lower optimisations but didn't have the higher opt levels.

OTOH I'm very pleased by how its running on this particular platform.

Anonymous said...

So that's how we got fossils: God increased the resolution, mountains sprang up and any creatures unlucky enough to find themselves inside them were turned to stone!