Sunday, May 6, 2012

3889.txt

cc: p.jones@uea.ac.uk, i.harris@uea.ac.uk, t.osbornatXYZxyz.ac.uk
date: Tue, 22 Apr 2008 12:54:53 +0100 (BST)
from: P.JonesatXYZxyz.ac.uk
subject: Re: [Fwd: Re: {Spam?} Re: Status]
to: "David Lister" <D.ListeratXYZxyz.ac.uk>


David,
Looks as though the time series data are wrong then
for 61-96 assuming that you've not updated the earlier data
with VAP data, where the earlier was something other than VAP.

As I said earlier the time series data for the Gabon sites
for VAP look like temperature data from somewhere like Ireland.

Hope you can sort this out!

Cheers
Phil


> Phil/Harry/Tim,
>
> I have looked in Mark New's normals database (I have not yet looked in the
> "red book"). The Gabon stations are listed below:
>
> 6450000GAB8795 1719 301 308 302 309 304 274 248 250 268 288 297
> 306
> 6450300GAB6190100-1 287 289 296 308 294 242 223 232 254 277 287
> 277
> 6450400GAB6190100-1 298 301 305 301 301 279 255 261 286 287 294
> 289
> 6451000GAB8795 1419 272 256 255 260 263 256 247 243 249 260 261
> 258
> 6454500GAB6190100-1 239 241 247 250 249 242 226 226 240 244 243
> 257
> 6455000GAB8795 1119 301 299 294 301 305 274 249 250 267 283 290
> 301
> 6455200GAB6190100-1 261 264 266 253 268 254 237 236 249 254 258
> 249
> 6455600GAB6190100-1 251 249 253 255 250 239 227 226 239 244 243
> 240
> 6456500GAB6190100-1 227 263 268 268 260 230 219 226 248 261 263
> 258
>
>
> These values look reasonable to me. I will check further if required.
> NB four of the above (code="-1") are derived variables. The others come
> from CRU's GPD.
>
> Cheers
>
> David
> On Tue, 22 Apr 2008 P.JonesatXYZxyz.ac.uk wrote:
>
>> ---------------------------- Original Message
>> ----------------------------
>> Subject: Re: {Spam?} Re: Status
>> From: f028atXYZxyz.ac.uk
>> Date: Tue, April 22, 2008 7:45 am
>> To: "Ian Harris" <i.harrisatXYZxyz.ac.uk>
>> Cc: "Tim Osborn" <t.osbornatXYZxyz.ac.uk>
>> "Phil Jones" <p.jonesatXYZxyz.ac.uk>
>> --------------------------------------------------------------------------
>>
>>
>> Tim, Harry,
>> First can't see why Harry's email has come as spam.
>>
>> A few thoughts on the data for the 2 Gabon stations.
>> If these are supposed to VAP data then the data for the
>> period 1961-1996 looks completely wrong. The later data looks
>> better!
>>
>> Seems as though these two sites and the others for the country
>> ought to be removed. I'm making this suggestion based on
>> common sense. Gabon is near the equator and I think most is
>> jungle.
>>
>> I could be wrong. The Sudan VAP for El Dueim looks OK. It should
>> drop during teh dry season.
>>
>> Maybe ask David to have a look. Or have a look at a station in an
>> adjacent country.
>>
>> No idea what the Gabon numbers are for the 61-96 period! They look
>> like temperatures from somewhere like Ireland!
>>
>> Hope you can find the problem!
>>
>> Cheers
>> Phil
>>
>>
>>> Hi Tim,
>>>
>>> A little more detail now I've had a chance to look at the files. Try
>>> and view this email in a monospaced font! Or it'll look awful.
>>>
>>> On 21 Apr 2008, at 16:23, Tim Osborn wrote:
>>>
>>>> Hi Harry,
>>>>
>>>> yes, please focus on WET first.
>>>>
>>>> then please take a look at the VAP station data in the countries of
>>>> Gabon (1st priority) and Sudan (2nd priority), for 2003 onwards.
>>>>
>>>> I've checked the synthetic vap and that does not have extremely high
>>>> recent values, so it seems to be coming from the stations. I masked
>>>> out areas not near to stations and the high extremes are still
>>>> there... so that's further evidence it is in the stations. Gabon
>>>> shows particularly strong anomalies (approaching 20 hPa above normal
>>>> at times), but there are some over 10 hPa above normal in the Sudan
>>>> area too.
>>>>
>>>> The two ncview.*.pdf files attached show the anomaly maps for 2
>>>> selected months during 2003-2005 (see file names for specific
>>>> months/years). I've masked in dark blue areas outside the CDD
>>>> distance from a station (according to your ...vap.stn.nc file). The
>>>> high values are red (surrounded a yellow stripe). Gabon in west
>>>> equatorial Africa stands out in both, Sudan in one. Both have
>>>> numerous other high anomalies like this from Jan 2003 onwards. They
>>>> are the cause of spikes in country-mean VAP in the other attached file
>>>> -- see page 15 for Gabon (and neighbouring countries) and page 13 for
>>>> Sudan (and neighbouring countries). Sudan looks less severe, but
>>>> that's partly because it is larger and the averaging takes in more
>>>> synthetic vap values from outside the station's neighbourhood.
>>>>
>>>> The extreme positive anomalies occur close to (this is from visual
>>>> reading off the screen, so not fully precise!):
>>>>
>>>> Gabon: 9.25 E, 0.25 S -- is there a station at Port-Gentil?
>>>
>>> There is - well, three, as this is a database I haven't cleansed:
>>>
>>> -645014 -187 1102 89 MOUILA GABON 1971 1996
>>> -999 -999.00
>>> -645009 50 940 15 LIBREVILLE GABON 1971 1996
>>> -999 0.85
>>> -645008 -70 875 4 PORT GENTIL GABON 1971 2002
>>> -999 -999
>>> -645007 -70 875 4 PORT GENTIL GABON 1994 2003
>>> -999 nocode
>>> 6450000 45 942 15 LIBREVILLE/LEON MBA GABON 1979 2007
>>> -999 -999
>>> 6450100 -70 875 4 PORT GENTIL GABON 2003 2007
>>> -999 0
>>> 6451000 208 1148 599 BITAM GABON 1971 2007
>>> -999 -999
>>> 6450300 -342 1065 34 MAYUMBA GABON 1971 2005
>>> -999 -999
>>> 6450400 100 960 13 COCOBEACH GABON 1971 2004
>>> -999 -999
>>> 6450700 -285 1102 79 TCHIBANGA GABON 1971 2007
>>> -999 -999
>>> 6451000 208 1148 599 BITAM GABON 1971 1996
>>> -999
>>> 6455100 -72 1023 26 LAMBARENE GABON 1971 2007
>>> -999 -999
>>> 6455200 78 1153 583 MITZIC GABON 1971 2004
>>> -999
>>> 6455600 57 1287 515 MAKOKOU/EPASSENGUE GABON 1971 2006
>>> -999
>>> 6456000 -83 1271 485 LASTOURSVILLE GABON 1971 1996
>>> -999 -999
>>> 6456500 -153 1326 573 MOANDA GABON 1971 1996
>>> -999
>>> 6455000 -187 1102 89 MOUILA CITY GABON 2003 2007
>>> -999 0
>>>
>>> Now, NONE of these stations should contribute to the product, since our
>>> specified cutoff is 75%, or 23 values per month between 1961 and 1990.
>>> But for a couple, a normals line is present:
>>> 6450000 45 942 15 LIBREVILLE/LEON MBA GABON 1979 2007
>>> -999 -999
>>> 6451000 208 1148 599 BITAM GABON 1971 2007
>>> -999 -999
>>>
>>> These two stations have several things in common. Firstly, they both
>>> have a populated normals line, even though their data counts are too
>>> low. Secondly, their values (and normals) are significantly below the
>>> rest of GABON (ie, between 40 and 180 rather than between 200 and 350).
>>> Thirdly, they both have a swathe of missing values 1997-2002, followed
>>> by patchy data that looks like the rest of GABON (ie, too high for the
>>> normals lines). Here they are, in full:
>>>
>>> 6450000 45 942 15 LIBREVILLE/LEON MBA GABON 1979 2007
>>> -999 -999
>>> 6190 65 65 75 87 111 135 149 147 135 110 83 72
>>> 1979-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 79 83
>>> 1980 59 80 77 85 99 136 149 160 149 100 76 71
>>> 1981 68 59 98 93 125 135 155 156 135 92 83 54
>>> 1982 51 60 73 78 107 147 161 150 149 114 94 70
>>> 1983 79 52 78 93 109 137 182 159 136 105 87 72
>>> 1984 69 62 66 78 109 121 139 154 131 111 95 74
>>> 1985 43 45 72 92 118 123 154 139 139 107 68 83
>>> 1986 64 37 71 82 115 140 146 130 106 113 91 77
>>> 1987 45 65 63 100 100 138 150 149 141 106 83 74
>>> 1988 80 67 76 84 120 136 150 148 134 114 86 93
>>> 1989 76 77 92 84 121 129 156 147 144 116 79 76
>>> 1990 79 85 89 86 112 137 136 151 121 115 84 69
>>> 1991 64 55 97 84 100 119 164 150 138 107 80 68
>>> 1992 66 76 83 93 130 146 153 160 134 95 93 72
>>> 1993 77 61 64 93 110 127 138 127 122 93 64 76
>>> 1994 72 64 91 89 112 133 173 150 137 104 115 84
>>> 1995 71 85 69 95 105 132 178 152 132 130 86 58
>>> 1996 59 60 69 86 101 138 147 153 114 106 89 58
>>> 1997-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1998-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2001-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2002-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2003 305 320 311-9999-9999 275 277-9999-9999 300 303 313
>>> 2004 319 315-9999-9999 290 265-9999-9999 283-9999-9999 307
>>> 2005 316 318 311-9999 291 258-9999-9999 281-9999-9999 299
>>> 2006-9999-9999-9999-9999-9999 295 266-9999-9999-9999-9999-9999
>>> 2007 318 313-9999-9999-9999 277-9999-9999-9999-9999-9999-9999
>>>
>>> 6451000 208 1148 599 BITAM GABON 1971 2007
>>> -999 -999
>>> 6190 67 65 74 84 110 135 149 147 133 111 84 72
>>> 1971 66-9999 65 85 122 128 157 149 117 104 79 81
>>> 1972 62 72 73 88 105 121 162 142 114 91 85 70
>>> 1973 69 69 70 78 113 140 150 154 137 100 79 68
>>> 1974 80 72 83 81 106 134 140 151 127 88 85 90
>>> 1975 85 64 76 90 101 129 158 161-9999 100 80-9999
>>> 1976 73 69 61 72 104 132 142 131 128 116 86 60
>>> 1977 70 78 84 77 100 136 149 151 120 120 85 80
>>> 1978 66 62 80 80 117 129 140 132 127 113 82 69
>>> 1979 44 53 74 82 108 143 144 140 127 116 85 83
>>> 1980 60 79 75 77 91 128 145 155 141 101 78 70
>>> 1981 69 59 97 86 120 130 154 159 134 102 90 59
>>> 1982 62 65 72 77 110 150 165 156 150 115 95 71
>>> 1983 82 55 81 91 105 138 181 160 133 107 86 73
>>> 1984 70 63 65 71 106 125 140 154 134 120 100 74
>>> 1985 46 47 70 88 118 124 147 137 137 106 68 83
>>> 1986 66 34 70 77 111 144 147 130 111 119 92 77
>>> 1987 46 66 64 95 98 139 152 153 150 112 87 77
>>> 1988 84 69 78 84 121 135 148 147 130 115 84 92
>>> 1989 77 71 82 76 114 119 145 138 141 116 76 75
>>> 1990 76 81 82 82 107 131 134 150 120 115 86 70
>>> 1991 66 53 88 75 94 117 162 150 137 107 81 68
>>> 1992 65 74 81 84 116 146 166 165 133 89 92 71
>>> 1993 77 63 70 100 123 140 147 135 131 101 68 80
>>> 1994 77 65 83 86 115 133 170 153 136 102 114 85
>>> 1995 71 84 70 96 109 134 178 157 131 134 88 60
>>> 1996 62 58 66 85 103 134 140 149 121 111 83 58
>>> 1997-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1998-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 1999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2000-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2001-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2002-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999
>>> 2003 255-9999 261-9999-9999 253 250-9999-9999 256 255 259
>>> 2004 265 250-9999-9999 259 245-9999-9999 245-9999-9999 266
>>> 2005 266-9999-9999-9999 257-9999-9999-9999-9999-9999-9999-9999
>>> 2006-9999-9999-9999-9999-9999 253 247-9999-9999-9999-9999-9999
>>> 2007 244 252-9999-9999-9999 253-9999-9999-9999-9999-9999-9999
>>>
>>> I would say that somehow the presence of an illegal normals line is
>>> allowing these 'bad' stations to be included. You'd almost call it
>>> sabotage, wouldn't you?
>>>
>>> The obvious fix would be to replace the normals with missing values,
>>> though it rankles to leave bad data in the database. And we'd end up
>>> with no Gabon stations contributing at all!
>>>
>>>> Sudan: 32.75 E, 13.25 S -- perhaps Khartoum or somewhere just to the
>>>> south of that?
>>>
>>> It's a similar story to Gabon, sorry. This is the culprit, showing
>>> header, normals line, and 2003-2007:
>>>
>>> 6275000 1400 3233 378 ED DUEIM SUDAN 1971 2007
>>> -999 -999
>>> 6190 64 60 69 78 102 127 143 143 130 109 83 70
>>> 2003 89 69 63 85 213 210 247 266 247 198 119 102
>>> 2004 87 76 74 86 126 192 212 237 231 168 117 103
>>> 2005 77 107 69 97 125 186 231 247 233 176 108 122
>>> 2006 105 87 103 73 173 179 227 246 239 269 97 83
>>> 2007 80 70 82 95-9999 203-9999-9999-9999-9999-9999-9999
>>>
>>> (again, there was a block of missing values 1997-2002).
>>>
>>> The only other Sudanese stations with enough data to contribute are OK:
>>> 6260000 2182 3148 183 HALFA (WADI) SUDAN 1971 2007
>>> -999 -999
>>> 6265000 1917 3048 226 DONGOLA/DUNQULAH SUDAN 1971 2007
>>> -999 -999
>>>
>>> In this case, I'd say the 2003-2007 additions to Ed Dueim should be
>>> deleted. Now if we just had a data format with full historical
>>> metadata..
>>>
>>>> I presume you can set these to missing from Jan 2003 onwards? Or just
>>>> set the outliers to missing? cc'd to Phil for his suggestion... we
>>>> need something quick and easy, but defendable! They are so extreme
>>>> relative to the standard deviation that these two particular ones
>>>> cannot really be left as they are.
>>>>
>>>> There are also many other positive and negative vap anomalies
>>>> (especially post 2003, but also some pre 2003) that seem dubious (5 to
>>>> 10 hPa, compared with the vap standard deviation which is not far from
>>>> 1 hPa in most places (well, 0.5 to 2 hPa). I haven't bothered to list
>>>> the locations/years/months of these strange values because they
>>>> typically last only 1 month and they are mostly in regions of the
>>>> world with more dense observations and hence spatial averaging tends
>>>> to reduce the anomalies rather quickly. The reason for picking out
>>>> the Gabon and Sudan ones is that they seem to be affecting many months
>>>> and also in regions with less dense networks and hence 1 station
>>>> severely affects an entire national-average (and indeed neighbouring
>>>> countries too). However I've cc'd this to Phil to get his opinion on
>>>> how much time should be spent on these issues.
>>>
>>> I'd say that if these countries are anything to go by, the VAP database
>>> needs a spring clean!
>>>
>>>> Note that it isn't all bad news :-) There were many dubious looking
>>>> spikes in CRU TS 2.1 vap (even in "well-observed" countries like Spain
>>>> and Portugal) that have improved in CRU TS 3.0 vap! (see the attached
>>>> time series).
>>>
>>> Cool.
>>>
>>>> Cheers
>>>>
>>>> Tim
>>>>
>>>> P.S. I've just realised something... shouldn't these anomalies have
>>>> been ignored if they are > 3 (or is it 4) S.D. from the normal? Or
>>>> maybe that option isn't used for vap?
>>>
>>> As I stated previously - the 3 SD limit's applied by the anomaly
>>> program and so will be in force for both 'real' and synthetic vap.
>>>
>>> Cheers
>>>
>>> Harry
>>>
>>>
>>>>
>>>> At 15:09 21/04/2008, you wrote:
>>>>> Hi Tim,
>>>>>
>>>>> WET almost done; should be finished tonight (will complete it
>>>>> remotely). Note that, if you're investigating it, WET uses a
>>>>> percentage anomalies, I presume this is because the incoming
>>>>> synthetics are PRE-derived.
>>>>>
>>>>> FRS is also within reach, but has a lower priority.
>>>>>
>>>>> Cheers
>>>>>
>>>>> Harry
>>>>> Ian "Harry" Harris
>>>>> Climatic Research Unit
>>>>> School of Environmental Sciences
>>>>> University of East Anglia
>>>>> Norwich NR4 7TJ
>>>>> United Kingdom
>>>>>
>>>> <ncview.vap.dec2004.pdf><vap_cty.pdf><ncview.vap.sep2005.pdf>Dr
>>>> Timothy J Osborn, Academic Fellow
>>>> Climatic Research Unit
>>>> School of Environmental Sciences
>>>> University of East Anglia
>>>> Norwich NR4 7TJ, UK
>>>>
>>>> e-mail: t.osbornatXYZxyz.ac.uk
>>>> phone: +44 1603 592089
>>>> fax: +44 1603 507784
>>>> web: http://www.cru.uea.ac.uk/~timo/
>>>> sunclock: http://www.cru.uea.ac.uk/~timo/sunclock.htm
>>>>
>>>
>>>
>>
>>
>>
>>
>


No comments:

Post a Comment