Wednesday, May 9, 2012

4060.txt

cc: Edward Cook <drdendroatXYZxyzo.columbia.edu>
date: Fri, 13 Jun 2008 12:08:32 -0400
from: Edward Cook <drdendroatXYZxyzo.columbia.edu>
subject: Fwd: Some PR Challenge comments
to: Keith Briffa <K.BriffaatXYZxyz.ac.uk>

Just what I sent Casper. FYI only.

==================================
Dr. Edward R. Cook
Doherty Senior Scholar and
Director, Tree-Ring Laboratory
Lamont-Doherty Earth Observatory
Palisades, New York 10964 USA
Email: [1]drdendro@ldeo.columbia.edu
Phone: 845-365-8618
Fax: 845-365-8152
==================================

Begin forwarded message:

From: Edward Cook <[2]drdendro@ldeo.columbia.edu>
Date: June 13, 2008 10:46:43 AM EDT
To: Caspar Ammann <[3]ammann@ucar.edu>
Cc: Edward Cook <[4]drdendro@ldeo.columbia.edu>, Nicholas Graham <[5]ngraham@hrc-lab.org>,
"Rosanne D'Arrigo" <[6]druidrd@ldeo.columbia.edu>
Subject: Some PR Challenge comments

Hi Casper,
As you are aware, I have some serious concerns about how the PR Challenge has been
formulated. So as of now, I have no intention of participating in it. I mentioned to you in
Trieste that I thought the better way to go was to use real climate data and real proxies
for a range of reconstruction problems (temperature, precipitation, drought, pressure on
local, regional, hemispheric scales) such that TRUE uncertainties in the reconstructions
and the methods used could be objectively and realistically evaluated. The work Keith,
Phil, and I did for NATO is a very useful template for doing so and the results can be
objectively evaluated by setting up the testing and comparison procedures in an a priori
way. However, I also recognize the value of pseudo-proxies in this context and support the
original PR Challenge idea of using model output to produce pseudo-climate and pseudo-proxy
data for testing as well. Indeed, one could create the pseudo-data sets to match fairly
closely the properties of the actual data sets chosen for testing. Including both real and
pseudo data in the experiments would dramatically strengthen your PR Challenge in my
opinion because it would reduce the dependence of the results and their interpretations on
the considerable uncertainty that still remains in how to generate pseudo-proxy data using
forward-modeling methods. Tree rings are most developed here for forward modeling I think
(e.g. the Vaganov-Shashkin model), but to assume that the V-S model can magically model the
tremendous complexity of growth responses to climate in tree rings at the genus and species
levels is simply wrong. Maybe the V-S model will get a lot of it right, but it will likely
be in a highly idealized way that will very likely under-estimate the true uncertainty in
tree-ring reconstructions. I am also not sure that useful forward models for other proxies
like corals even exist yet.
Perhaps the biggest complaint I have with the PR Challenge is its "double-blind" design
that gives the tester no information on the problem being tested even though no such
situation ever occurs in practice. At that level alone, the "double-blind" design used in
the PR Challenge is very odd. Regardless, I fully understand the value of "double-blind"
statistical experiments as the only way to test for true causality in, for example, cancer
drug tests. But this is simply not the kind of problem we are dealing with in the proxy
reconstruction game (with all due respect to your statistician colleagues). We are much
more involved in something more akin to epidemiological hypothesis testing on real
pre-existing data, like the association between smoking and lung cancer in humans. Do
medical epidemiologists perform "double-blind" experiments to determine if smoking causes
lung cancer? Basically no. They use the data available to them to answer that hypothesis
with a high degree of statistical certainty without the need to conduct "double-blind"
experiments on humans (not possible for smoking and lung cancer in any case). So in my
opinion, the "double-blind" approach is unnecessary and, I would argue, even inappropriate
here. I also don't like it because that is not the way we should be conducting our science.
When I worked with Keith and Phil for a month at CRU on our NATO test data sets and
reconstruction methods, we worked in a completely objective and open way that enabled us to
debate various options and interpretations of our programs and reconstructions. We were
also able to thoroughly test our programs using identical data for the special case of full
MLR. This enabled us to be absolutely certain that we could compare results in the various
best-subset cases that are typically used for reconstructing climate from tree rings. There
does not appear to be any such mechanism for doing so in your PR Challenge, so differences
found may be as much related to odd programming matters rather than the methods being
compared. In any case, comparison of reconstruction methods should be conducted openly and
that is the way that I love to work. So if the "double-blind" requirement were removed from
the PR Challenge, I would be much more inclined to participate. I am not adverse to being
proven wrong (or "less right" perhaps) in an open environment that allows for direct
"give-and-take" on the merits of the cases being tested and argued. A "double-blind"
approach is explicitly designed to eliminate the scientific openness needed to evaluate and
debate reconstruction methods in my opinion. At that level, and for reasons stated above, I
reject it.
Regards,
Ed
==================================
Dr. Edward R. Cook
Doherty Senior Scholar and
Director, Tree-Ring Laboratory
Lamont-Doherty Earth Observatory
Palisades, New York 10964 USA
Email: [7]drdendro@ldeo.columbia.edu
Phone: 845-365-8618
Fax: 845-365-8152
==================================

No comments:

Post a Comment