A Parallel, Utility-Based Computing Approach To Bathymetric Navigation Surface Creation From Heterogeneous & Distributed Bathymetric Sounding Databases.
I've often had to deal with truly large bathymetric sounding databases. With increasing use of multi-beam echo sounders (MBES) bathy databases are just getting larger and larger and often contain many years of data of varying qualities. What happens if you want to make use of all this data to create new bathymetric products? You need to account for the various uncertainties in quality, temporal variation, accuracy etc.Modern processing algorithms such as CUBE (Combined Uncertainty Bathymetric Estimation) attempt to account for these types of uncertainties and can derive bathy products such as "safe navigation surfaces" from such heterogeneous data.
When these databases get very large - or indeed if they are distributed across multiple instances or sites it becomes very difficult to load and process all of the input data. This got me to thinking about using parallel processing techniques (HPC - High Performance Computing etc) to undertake and speed up such tasks. I reviewed many parallel processing topologies and came to the conclusion that for the occassional processing job a dedicated compute cluster would not be necessary. Why not just make use of the many thousands of computers that are essentially idle in the office when everyone goes home? Just like the SETI project or a loosely coupled Beowulf cluster (also known as a Utility cluster or a Network of Workstations [NOW]).
Well I got to fully designing a modified CUBE algorithm (I think its an improvement), data flows, object models, data models etc and I began writing the code for it all. After a lot of effort (and a 41 page proposal document) on my first test run I realised that the Control Database was going to be a bottleneck so I started a complete re-design which I will post at some stage in the future when its all developed a bit more.
