from: Tim Mitchell <tim.mitchellatXYZxyzrey.ac.uk>
subject: Re: missing, constant, quick-and-dirty
to: Timothy Carter <tim.carteratXYZxyzaristo.fi>, Mike Hulme <m.hulmeatXYZxyz.ac.uk>, Phil Jones <p.jonesatXYZxyz.ac.uk>
One point of clarification, and then a couple of ways to make life easier...
1. We did indeed use estimates of secondary variables (vap,cld) from the
primary variables (tmp,pre,dtr) in order to provide as much information as
possible. Mark New developed the relationships in the 2000 paper, and I
followed his method in subsequent observed grids. So any further work in
this direction will need to improve Mark's methods, and it may prove
difficult to substantially improve coverage by this method.
2. I did try to produce some reasonably comprehensive documentation on how
the climate data-sets might be appropriately used by impacts people. It took
the form of a draft paper written in February 2003. I intended to take this
forward with Tim C, but it got lost in a black hole somewhere between UEA
and SYKE. I remember doing something with Mike H about turning this into a
Tyndall working paper, but I can't remember whether this happened.
I had some exchanges with Tim C over this putative paper before leaving, and
I suggest that some of the material from this document could still be used
in future. Either a full-blown research paper on how to use such climate
data-sets could be produced, perhaps under the ATEAM banner (a special
issue?), or something with a lower profile. The general aim would be to
expose some of these issues to the light of day, and suggest some practical
solutions for future experimental design.
3. Mike's version B is already available, and version A is easily obtained
using the public data. CRU TS 1.2 (Europe 10') is complemented by CRU TS 2.0
(globe 0.5deg), produced by the same methods on a coarser grid. For CRU TS
2.0, at the request of users, I provided the information necessary for
version A. See
Files are available that parallel the data files. For every datum in CRU TS
2.0, there is a parallel datum that counts the number of observed values
within the correlation decay distance for the variable. See that webpage for
For a quick'n'dirty solution, assign each 0.5deg count to the nine 10'
grid-cells within that cell, and you have an equivalent for CRU TS 1.2.
Where the count does not exceed 0 (or 1 perhaps?), insert missing values in
the data grids.
But bear in mind that this is not a full solution, because I still hold to
my assertion that there is a continuum between 'full representation' and 'no
representation' of actual climate variations, not a black/white boundary.
Using these station counts in anger will probably demonstrate that!
On 1/4/04 7:42 am, "Timothy Carter" <tim.carteratXYZxyzaristo.fi> wrote:
> Dear all,
> Thanks Mike for seeing a way forward. We are currently examining a quick
> and dirty method for ATEAM purposes only. Obviously, if a surrogate set was
> to be provided online, we might like to spend a bit more time developing
> it. Obviously, CRU people have far more experience and access to data to
> come up with methods of doing this than we do here at SYKE (and, naturally,
> no resources!). Anyway, I will keep you posted on our initial attempts.
> At 17:32 31/03/04, Mike Hulme wrote:
>> Dear Tim^2 and Phil,
>> Ah, the good old days of discussions and arguments about data, scenarios
>> and impacts. Much more stimulating than most of the management stuff I
>> have to deal with these days.
>> I've now read the various comments and I agree with everything (of
>> course). Since Tyndall Centre is solution-oriented, how about a solution
>> (the impracticality of it concerns who can implement it given Tim M has
>> left, ATEAM money is spent, and ATEAM is virtually completed!):
>> To make the problem immediately obvious to people there should be three
>> versions, labelled in some such way:
>> Version A (MISSING): the dataset with missing values for where no observed
>> data within a certain radius for different variables/times
>> Version B (CONSTANT): the version now on release, i.e. relaxing the
>> problem areas/times to 1961-90 climatology
>> Version C (QUICK_AND_DIRTY): the problem areas and variables filled in
>> using some regression technique suggested by Tim C.
>> Impacts modellers are then forced to choose which one they use and in
>> making the choice they are brought face to face with the problem.
>> [As an aside I thought Mark developed such simple relationships for CLD
>> and DTR, or maybe this was just for normals and not for IAV].
>> At 14:52 31/03/2004 +0300, Timothy Carter wrote:
>>> Dear Phil,
>>> I agree with the thrust of your argument here. Yes indeed, impacts people
>>> rarely have much of a conception about the limitations of climate data,
>>> but I would say that this is particularly true where they wish to apply
>>> scenario information. They quite often expect to be provided with
>>> scenario information at the same spatial and temporal scale as the
>>> climate inputs for their models.
>>> I usually suggest to impact modellers who ask that they should first get
>>> their observational data in order before worrying about the scenarios.
>>> That is a obvious prerequisite for effective impact studies. The impacts
>>> observed in the recent past should be reproducable based on the climate
>>> observed during the same period. Some impact studies fall short even of
>>> this basic validation step.
>>> If they have reasonable quality high resolution observed data (which may
>>> be the case for individual sites or even limited regions) then how they
>>> perturb this for developing scenarios of the future climate is as much an
>>> art as a science. It is also worth noting that analysts sometimes use
>>> weather generators to represent present and future climate. Often, close
>>> inspection reveals a poor representation by WGs of the observed climate,
>>> in which case baseline impacts are estimated erroneously, even before
>>> considering future climate. So, again, the baseline climate is key.
>>> I recognise that one of the ATEAM ambitions was to have European coverage
>>> in the climate time series. This is fairly straightforward for future
>>> scenarios from GCM outputs. Unfortunately, it is not straightforward for
>>> representing the historical climate, as you at CRU can attest after 30
>>> years of working with such data.
>>> I am not criticising the climatological data that Tim et al. have
>>> prepared - it is the best one could have hoped for. The problem has been
>>> that the drawbacks of the data were not effectively communicated (I take
>>> equal responsibility for this). The data set was presented as a package -
>>> very convenient to download and apply, but not very easy at all to modify
>>> by non-climatologists according to user needs. The ATEAM impacts people
>>> knew that their models are sensitive to climate variability, so they were
>>> delighted to be offered this feature in the data set. What they didn't
>>> realise, was that the data sets are actually incomplete, although they
>>> appear not to be, by having values allocated at all grid boxes.
>>> So how are they to adjust their analysis to cover only the more reliable
>>> regions and time periods in the record? Some of the models being used in
>>> ATEAM are transient, so the effects of climate variability are cumulative
>>> over time - trees grow; species succeed one another according to tree
>>> mortality and ambient climatic variability. It isn't possible to run
>>> these models for 20 years in the 20th century and compare with the same
>>> period in the 21st century as it might be for e.g. hydrological models.
>>> Other methods are required to create a realistic time series over periods
>>> of hundreds of years.
>>> It is these issues that we have regrettably overlooked in providing these
>>> data. In this case, I do not blame the impacts people. In fact, I am
>>> grateful to them for highlighting some obvious difficulties in providing
>>> climate data for application. Yes, the problems are documented somewhere
>>> (as Tim points out), but how many ecologists have time or expertise to
>>> find the relevant climate journals and to interpret the subtleties of the
>>> many methods used to generate these observed data?
>>> I think the lesson to be learnt is that these data sets need to have up
>>> front (at the site of downloading) documentation that provides basic
>>> information on applications for which the data are or are not
>>> appropriate. This requires second guessing some of the potential
>>> applications, and though we already tried to do that in ATEAM we only
>>> partly succeeded. With only T and P, I doubt if anyone would have noticed
>>> any weaknesses in the data sets (they are reasonably complete). It was
>>> only because we were ambitious in introducing other variables, that the
>>> problems emerged.
>>> I suppose this dialogue process takes time, and we learn from our mistakes.
>>> I have seen Tom's (amazingly conciliatory) review. You should frame it in
>>> the Unit!
>>> Best regards,
>>> Tim C.
>>> At 12:18 31/03/04, Phil Jones wrote:
>>>> Tim C.
>>>> Quickly reading your response to Tim. M. I think you're defending
>>>> impacts analysts
>>>> far too much. Whenever I meet some of these people, I have to bite my
>>>> lip to avoid
>>>> saying something I'll regret. Impacts people need to be made aware of
>>>> the limitations
>>>> of observed data and even more of model data. What Tim has done is
>>>> likely the best that
>>>> can be done given the limitations of what we can get hold off, yet
>>>> still trying to maintain
>>>> the weak correlations between variables.
>>>> At many meetings impacts people ask for model futures for
>>>> variables and time intervals
>>>> we just don't have in the real world. How then do they test their
>>>> models? Chris Kilsby
>>>> is working to derive 5 minute rainfall scenarios for an EPSRC project,
>>>> because the
>>>> hydrologists on one project want this. There is one raingauge in the
>>>> UK with 5 minute
>>>> rainfall for 20 years. They want it for urban catchments in northern
>>>> England, the long
>>>> record is from Farnborough. When pushed on this they gave us one
>>>> year's data for site
>>>> near Bradford. They said they had techniques for making 1000 years of
>>>> records from
>>>> one year of data. Despite this being a climate change project they
>>>> just thought that
>>>> high-frequency rainfall variations will change according to the mean.
>>>> To show them at our next meeting, we're going through HadAM3P/H and
>>>> looking at convective/total precip and large-scale/total precip ratios
>>>> and A2 scenario
>>>> changes. I've never seen these sorts of plots before. The results are
>>>> frightening. In winter
>>>> over the Mediterranean, 90% of the rainfall over the sea is
>>>> convective, but on land less
>>>> than 10% is convective. I've never seen a variable delineate the
>>>> coastline so well. How does
>>>> large-scale rainfall which falls on the land not fall into the sea.
>>>> Tim may not have said, but we already have one review of the J.
>>>> Climate paper
>>>> (from Tom WIgley) which is by Tom's standards good. I'm dreading
>>>> getting the reviews back
>>>> as I think it will be me who has to respond to them. I know I'm not
>>>> going to have much
>>>> time to respond, so the first thing I'll do will be to ask for an
>>>> extension of the likely 1 month
>>>> that we'll be given - if the other reviews are as favourable as Tom's.
Dr. T. D. Mitchell --- 07906 922 489