Queuing and Scheduling

Useful HPC commands/links

Find out what queues are available:

qconf -sql

Find out information about a named queue

qconf -sq serial.q

Number of slots usually equals number of cores

Good intro
http://talby.rcs.manchester.ac.uk/~ri/_linux_and_hpc_lib/sge_intro.html

Translation between PBS and SGE
http://wiki.ibest.uidaho.edu/index.php/Tutorials:_SGE_PBS_Converting

Using environment vars/arguments
http://stackoverflow.com/questions/9719337/how-to-properly-pass-on-an-environment-variable-to-sun-grid-engine

Using python to poll the status of a job:

    cmd = 'qstat -f | grep %s' % job_id
    proc = subprocess.Popen([cmd], stdout=subprocess.PIPE, shell=True)
    output = proc.stdout.read()
    if output==None:
        return output
    
    else:
        status = output.split()[4].strip()
        return status


Explantion of queue error state E:

http://www.gridengine.info/2008/01/20/understanding-queue-error-state-e/

Compiling WPS

To compile WPS, had to modify configure.wps to indclude:

COMPRESSION_LIBS        =  -L/home/slha/usr/lib -ljasper \
                           -L/usr/lib -lpng12
COMPRESSION_INC         = -I/home/slha/usr/include

How many nodes?

Until now I didn't know how many cores I had to work on. The info in /proc/cpuinfo just didn't seem to tie in with what the vendor had supplied.  Then the penny dropped, /proc/cpuinfo only tells you about the current node, while I wanted to find out about the whole cluster. OK, seems obvious now, but here is some useful info I collected along the way.

One thing I haven't found is how to find out how many nodes (hosts) are in your local domain?

I'm adopting the following convention
1. A "chassis" houses one or more nodes.
2. A node contains one or more sockets.
3. A socket holds one processor.
4. A processor contains one or more (CPU) cores.
5. The cores perform FLOPS.

Command

Description

$cat /proc/cpuinfo

List processor information. Each physical chip has a unique ""physical id". Cores of the same chip have the same physical ID but different core ids. see http://www.richweb.com/cpu_info This will only tell you about the local node! You will need to SSH into other nodes to work out about them as well. Useful flags lm 64 bit support vmx Intel Virtualization Technology support ht indicates Hyperthreading support

Useful AWK script here:
http://superuser.com/questions/388115/interpreting-output-of-cat-proc-cpuinfo?rq=1

The head node on my cluster has two sockets, with quad core chips, but with hyperthreading enabled, so that it looks like 16 logical 'processors'.  The calculation nodes also have two sockets, with quad core CPUS, so 8 cores per node. In total I have 216 calculation cores to play with!

Renegade Master

Back once again, pre-empting a New Year resolution to re-start my work blog.  Lots of interesting stuff happening, or about to happen, in terms of wind power forecasting for offshore wind farms.

All I will say at the moment is: we're confident we can beat ECMWF at their own game. Watch this space. (nobody else is!).

ERA-Interim

ERA-Interim data is available for free for academic and commercial work:
http://www.ecmwf.int/research/era/do/get/index

Getting ECMWF data

So, you've heard 'ECWMF is far better than  NCEP', and you want a piece of some ECMWF data action. Where do you get get it? How do you find it?.  Here's a list to help you (me) out.

There are multiple routes to ECMWF data heaven. The most flexible and powerful, but most complicated is MARS. To use MARS fully you need the client, a piece of open-source software covered by a GPL licences, but costs £100. It is used to communicate with the data servers at ECMWF, and can subset data on the server side to reduce the volume of data needed to be transferred.  Quicker way to MARS is via  Web MARS.

Some links on MARS
www.ecmwf.int/services/computing/training/material/mars.pdf
www.ecmwf.int/publications/manuals/mars/practice/
www.ecmwf.int/publications/manuals/mars/practice/
www.ecmwf.int/services/archive/

The other route is the ECMWF data server. This gives you straight access, no software needed, no (not many) questions asked. It is actually just a linux server sitting outside the ECMWF firewall running MARS, but only allowing access to the disk archive, and not the tape archive.

To grab a dataset requires a request. Some fields used in the request are:

class	ECMWF classification (od, rd, e4, …)
stream	originating forecasting system(oper, wave, enfo, seas, …)
expver	version of the experiment (01 operational, 11, aaaa)
domain	area covered by the data (Global, Mediterranean, …)
system	seasonal forecast operational system (1, 2, 3)
type	type of field (an, fc, …)
levtype	type of level (pl, ml, sfc, pt, pv)
levelist	levels for the specified levtype (off if levtype=sfc)
param	meteorological parameter (t, temperature, 130, 30.128)
number	ensemble member (1, 2, …)
channel	brightness temperature frequency band
diagnostic	iteration sensitivity forecast products
frequency	direction 2-d wave spectra products
product	section,
latitude	longitude
ocean products

An example of a MARS request is:


retrieve,
class = ei,
stream = oper,
expver = 1,
date = 20071201/to/20071231,
time = 00/06/12/18,
type = an,
levtype = sfc,
param = sd,
target = “era-int.200712.sd”


The request is used to build a MARS tree, which describes the data hierachy to work through.

Truncation is done before interpolation to save resources, and is done depending on the final grid specified. The mapping between the final grid and the resolution truncated to is:

Grid increment Truncation

2.5  ≤ Δ T63

1.5  ≤ Δ < 2.5 T106

0.6  ≤ Δ < 1.5 T213

0.4  ≤ Δ < 0.6 T319

0.3  ≤ Δ < 0.4 T511

0.15 ≤ Δ < 0.3 T799

0.09 ≤ Δ < 0.15 T1279

0.0   ≤ Δ < 0.09 T2047

To subset an area use the boundaries North/West/South/East.

Archived data is available to all registered users. Real-time data only to the big spenders!

ECMWF model data

The global gridded datasets available from central meteorological agencies are amazing. They are the distillation of thousands of global observations from all kinds of sources, pulled together into physically consistent picture of the atmosphere using immensely advanced meteorological models. They are a triumph of the scientific method and the power of cooperation.

They are also confusing as hell to work with. The main problems are:

• multiple similar datasets, often different versions derived from others 
• different vertical coordinate systems
• multiple access routes to the same/similar data 
• no up-to-date description of the current system
• information has to pieced together from multiple, contradictory sources

People assume you're familiar with the models before you start: "Oh, you need the forecast from the T159L60 model". This post is, by definition*, an idiot's guide to ECWMF data.  I'll post this in two parts: the first describing the general system, and the second describing the practicalities actually getting and working with data.

* because it was written by me

ECMWF run the Integrated Forecast System. This is a spectral model: in 2006 the deterministic operational model ran at a resolution of a resolution of T799L91. A summary of this is

Numerical scheme	TL799L91 (triangular truncation, resolving up to wavenumber 799 in spectral space, linear reduced Gaussian grid. 91 levels between the earth's surface and 80 km), Semi-Lagrangian two-time-level semi-implicit formulation.
Time-step	12 minutes
Smallest wavelength resolved	50 km
Number of grid points in model	76,757,590 in upper air and 3,373,960 in surface and sub-surface layers
Grid for computation of physical processes is the Gaussian grid, on which single level parameters are available.	The average distance between grid points is close to 25km.
Variables at each grid point	wind, temperature, humidity, cloud fraction and water/ice content (also (recalculated at each time-step) pressure at surface grid-points), ozone

The spectral resolution refers to the highest wavenumber retained in the solution. A the relation between spectral and spatial resolution is given below.

Spectral	Gaussian	Lat/lon
T63	N48	1.875
TL95	N48	1.875
T106	N80	1.125
TL159	N80	1.125
T213	N160	0.5625
TL255	N128	0.7
TL319	N160	0.5625
TL399	N200	0.450
TL511	N256	0.351
TL639	N320	0.28125
TL799	N400	0.225
TL1023	N512	0.176
TL1279	N640	0.141
TL2047	N1024	0.088

As far as I understand it, the output of the spectral model is converted to reduced gaussian grid, using some kind of magic, very likely involving Fourier transforms.  A gaussian grid plays well with spectral models on a sphere, as the grid points are evenly distributed over the sphere.  This is then converted into a regular latitude-longitude grid (aka cylindrical equidistant aka Plate Carre aka unprojected) as this is what most sane people like to work with.

So remember: the end resolution oft quoted e.g. 1.0 degrees, 0.5 degrees, 0.25 degrees  is the resolution of the end product. It is possible to derive low-resolution datasets from a high resolution model and vice-versa, although obviously going to from a low resolution model to high resolution output will not magically add information.

The resolution above is used for the deterministic forecast.  This also is used to give the initial conditions of the control run of the ensemble forecast. 

The control run (unperturbed) of the ensemble is truncated to a lower resolution from the operational analysis.  In addition, 50 perturbed members make up the full 51-member ensemble.  The ensemble model is run in two legs, with a lower resolution after 10 days.

According to this news article, the resolution of the deterministic and ensemble system were upgraded on 26th January 2010 to the following scheme, which is around 16km resolution!

	Deterministic		Ensemble Prediction System (EPS)
			Leg A		Leg B / C
	Current	Upgrade	Current	Upgrade	Current	Upgrade
Spectral	T799	T1279	T399	T639	T255	T319
Reduced Gaussian grid	N400	N640	N200	N320	N128	N160
Horizontal grid resolution	~25 km	~16 km	~50 km	~30 km	~80 km	~60 km
Dissemination (LL)	0.25	0.125	0.5	0.25	0.5	0.5
Model Level Vertical resolution	91	91	62	62	62	62

In addition, there are multiple reanalysis datasets, most notably ERA-Interim.  This uses cycle 31r2, which is the system introduced in 2006. This is T255 spectral resolution, 60 vertical levels, and a reduced gaussian grid at approximately 79km spacing.

The ERA-interim reanalysis contains analyses at 00z, 06z, 12z and 18z, and 10 day forecasts from 00z and 12z.  Pressure level and surface level data are archived every 3-hours out to 240 hours. Model level data is archived out to 12 hours. Forecast fields are not available on isentropic or potential vorticity levels.

Another product is the multi-model ensemble, where analyses from different centres: ECMWF, UK Met Office, NCEP and DWD are used to drive an ensemble.