A Parallel Bathymetric Processing System

A Parallel, Utility-Based Computing Approach To Bathymetric Navigation Surface Creation From Heterogeneous & Distributed Bathymetric Sounding Databases.

I've often had to deal with truly large bathymetric sounding databases. With increasing use of multi-beam echo sounders (MBES) bathy databases are just getting larger and larger and often contain many years of data of varying qualities. What happens if you want to make use of all this data to create new bathymetric products? You need to account for the various uncertainties in quality, temporal variation, accuracy etc.

Modern processing algorithms such as CUBE (Combined Uncertainty Bathymetric Estimation) attempt to account for these types of uncertainties and can derive bathy products such as "safe navigation surfaces" from such heterogeneous data.

When these databases get very large - or indeed if they are distributed across multiple instances or sites it becomes very difficult to load and process all of the input data. This got me to thinking about using parallel processing techniques (HPC - High Performance Computing etc) to undertake and speed up such tasks. I reviewed many parallel processing topologies and came to the conclusion that for the occassional processing job a dedicated compute cluster would not be necessary. Why not just make use of the many thousands of computers that are essentially idle in the office when everyone goes home? Just like the SETI project or a loosely coupled Beowulf cluster (also known as a Utility cluster or a Network of Workstations [NOW]).

Well I got to fully designing a modified CUBE algorithm (I think its an improvement), data flows, object models, data models etc and I began writing the code for it all. After a lot of effort (and a 41 page proposal document) on my first test run I realised that the Control Database was going to be a bottleneck so I started a complete re-design which I will post at some stage in the future when its all developed a bit more.

Agile Projects & the Geo Information Management Framework.

Way back before ESRI adopted a metadata system and even before the OGC (Open Geospatial Consortium) had developed the idea of "contexts" a couple of colleages of mine and I designed and built what we called "GIM" - the Geographic Information Management Framework.

I was the primary architect and project manager and together with my colleagues we designed, built and rolled-out a fairly complex system which included ArcGIS, ArcIMS integration as well as generic integration with web-sites and data services, spatialisation of flat databases to ArcSDE, application and systems management and structured specific business processes. The total build to client acceptance took us 6 months!

We were all 24-28 year olds at the time and we adopted the traditional programmer's approach of little sleep until we had finished the final build... We really did work bloody hard back then!

Even today some of the concepts and ideas that we developed are yet to be realised in standard commercial, enterprise GIS software systems and environments - and I truly believe that we had ground-breaking ideas. To this day (almost 7 years on) I am still proud of the technical abilities / dedication / determination and sheer hard work of the team and what we achieved and I would like to thank the main guys involved.

This link demonstrates what we built: https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxhcGFyc2VjYXdheXxneDoxNDNlODMyYTgzOTE4ZGY1

We used agile / extreme project management techniques - we had 3 main technical members (full time) and up to 8 other technical members. Our main primary stakeholder group was 10 people and the total user base was 300 - 500 users extending later to a group of more than 1000.

Agile project management methods proved extremely effective in this particular instance because we had an unsure / indifferent stakeholder group and a very competent technical project team.

The system was extremely successful and has been in place with the client (with minor or no modification) for at least the past 6 years. Replacements to components of the framework (using recently developed core ESRI technolog) still leverage the core model - hence proving the extensibility/scalability of the original solution.

Oil Spill Advection Modeling

This was one of the early development projects I did at the job I was in at the time. We were basically asked to develop in an ArcGIS environment a tool that takes wind and current data from a weather station database and predict the movement (advection) of an oil spill over time.

This is actually quite a simple task - the maths is pretty basic; you take a ratio of the wind and current vectors over a particular time step and this indicates the final motion vector of the spilled oil for the time step.

Two of us worked on this project and it took us 6 days from design to beta release.

The advection engine would later be used in combination with a "dispersion" model which I developed several years later.

Seismic Navigation Data Management

This was a quick interim build to solve a data migration problem. We had seismic navigation / shot point data etc in a large Oracle database which was maintained and loaded by a utility that was running on old Solaris UNIX boxes. We wanted to get rid of the old UNIX boxes and replace them with Windows machines. At the time we were meant to port everything to ArcGIS however back then the SDE feature model was not sufficiently matured to manage attributed shotpoint data with geometric and linearly referenced constraints to the corresponding seismic line. Our solution was to NOT migrate the Oracle database but instead put a quick Windows application on top of it that could manage the loading and editing.

After a few years we migrated the legacy database to SDE using linear referencing and thats how it remains to this day (to my knowledge).

ASCII to DGN Transforms

In 1999 I started working for a company which back then was called Fugro GeoSoft Solutions (in Singapore) - I think it's changed its name a couple of times now though. One of the projects I worked on was a GUI for an engine that they had developed called ATRANS. I don't think ATRANS exists anymore but when I joined it was used by a number of Oil and Gas companies to convert ASCII data to Microstation DGN format (Microstation 95 running on Sun Solaris UNIX workstations).

ATRANS was a C program built using MDL (the Microstation Development Language). It was like an early spatial ETL (Extract Transform and Load) tool - somewhat similar in concept to Safe Software's FME (Feature Manipulation Engine) product. You would specify using a FDF (feature definition format) file how the ASCII data should be read and transformed into a DGN file. It was a very powerfull and flexible tool for its time - we often used it to convert UKOOA P1/90 files to shot point plots in CAD.

As we were moving away from UNIX and more towards Windows NT my boss asked me if it would be possible to build a GUI for the FDF file creation (which was previously done in UNIX using VI)... it would mean that users did not need to be so familiar with the FDF file format which was a bit cumbersome. The result is shown in the screen capture above - the application parsed and generated FDF files into an object model and could also execute the FDF file against the ATRANS MDL by using a simple Shell command. It also used RTF to nicely colour code the FDF file elements - just like a proper IDE (Integrated Development Environment).

UGRS/MGRS Conversion

Back when I was at university I wrote a code library / application that converted Military or Universal Grid References (MGRS/UGRS) to any other coordinate system.

The software was developed as a uni project but the clients were AUSLIG (the Australian Land Information Group) and FESA (Fire and Emergency Services Authority) of Western Australia.

Looking back at it now I would have done things a bit differently - but you learn.

At the time Australia was moving away from the ANS (Australian National Spheroid) and associated AGD (Australian Geodetic Datum) and going towards a geocentric datum GDA 96/2000 (Geodetic Datum of Australia). The fire and emergency services authority in particular had lots of data referenced in a rather odd location reference system called UGRS (Universal Grid Reference System) which is based on UTM (Universal Transverse Mercator) but on ANS. UGRS is somewhat like the Maidenhead locator system used by radio amateurs, it uses a single string of characters and numbers to reference an approximate location.

56JMP973523 - For example is the UTM coordinate 497300 E, 6852300 N in UTM Zone 56

The clients wanted a way to convert the UGRS locations to UTM but on the new GDA datum and in turn output the new coordinates back to UGRS (on the new datum). They also had a desire to have the coordinate translation work inside ArcGIS 8 (which was about to be released - and was the talk of the GIS community at the time).

My solution was to implement a geodetic and map projection engine which could handle UGRS as a COM library (so that ArcGIS could make use of it) and also throw my own user interface on top of it. The result can be seen in the screen capture above.

It was a valuable learning experience for me - because not only did I finally get a chance to use COM properly and build a flashy GUI with my own vector display components - but it also made absolutely sure I understood the mathematics behind coordinate systems and datum transforms.