The dreaded GRIB format

I'm trying to do some processing of GFS forecast data, to try and find a nice way of interpolating wind speeds to heights above sea level (and eventually above ground level, but that may have to wait).

I thought I could use the Climate Data Operators (CDO), but my grib files appear to have duplicate records in them. Or they do according to wgrib2. To remove duplicate records you can do:

wgrib2 IN.grb -submsg 1 | unique.pl | wgrib2 -i IN.grb -GRIB OUT.grb


Note:
(a) for messages with submessages, only the inventory of the 1st message is checked. i.e. if the 1st submessage matches and the second submessage doesn't, the entire message is removed.
(b) order is preserved
(c) submessage structure is retained, see (a)
(d) note that on the command line, "GRIB" is capitalized

Where unique.pl is:

----------------------- unique.pl ------------------------
#!/usr/bin/perl -w
# print only lines where fields 3..N are different
#
while () {
chomp;
$line = $_;
$_ =~ s/^[0-9.]*:[0-9]*://;
if (! defined $inv{$_}) {
$inv{$_} = 1;
print "$line\n";
}
}
--------------------- end unique.pl ----------------------


UPDATE: This didn't help - it did seem to remove duplicate records, but CDO still didn't like the data. In any case I found out MET doesn't support grib2 format, so I need to convert my GFS files into grib1 in order to do any verification.

I did, however, find an awesome set of tools called gribmaster. Which will fetch grib data from loads of sources. This will be a godsend when I eventually try and make WRF operational, but even now it's very useful as it contains, pre-compiled, wgrib, wgrib2, and gribconv, plus some other tools for working with GEMPAK. In short it is awesome, and knows it.

So, my interim solution is to use 10m wind from GFS for validation against 80m towers (I know), but at least it lets me build the rest of the framework without wasting more time working with grib files. My strategy is this:


grib2 files ----> wgrib2 -----> 10m wind only ------> gribconv ------> grib1 files -----> met


Meteorologists were only ever interested in surface or upper air. The wind industry is interested in 80-100m. Hopefully one day they'll meet.