Death by Complexity

This post will be incomprehensible.

Over the last three days I have spent 4.5 hours working on GDAL Ticket 3276 related to a failure to use external overviews with JPEG2000 compressed NITF files - in particular for NITF files containing more than one jpeg2000 image.

This bug was particularly hairy because it comes at the intersection of several things that are messy/complex in GDAL:

1. JPEG2000 in NITF is implemented by creating JP2KAK driver dataset wrapping the jpeg2000 image data within the NITF file, and then using it's bands indirectly as bands for the NITF dataset. These band objects still mostly think of the jpeg2000 dataset as "their" dataset but for some purposes we really might wish they knew about the NITF dataset that appears to own them.
2. NITF files can contain more than one image. Such multi-image files are treated as containing subdatasets, one per image. For the most part these subdatasets are intended to act as freestanding things, but they are also, to some extent related back to the single file on disk containing the subdatasets.
   
3. Overviews in GDAL are mostly handled through an overview manager object embedded in the GDALDataset base class. However, JPEG2000 images have built-in overviews not handled through the overview manager.
   
4. The PAM (Persistant Auxilary Metadata) mechanism is used via an intermediate GDALPamDataset class to provide a way of storing additional information about datasets that the intrinsic format does not support. This information is stored in an .aux.xml file associated with the main data file.
5. In GDAL 1.7 a new capability was added to store PAM information for subdatasets in an .aux.xml file associated with the main data file so that subdatasets would work as much like a regular dataset as possible.
6. In GDAL 1.7 support was added for building overviews on subdatsets. Since normally overviews would be stored in a .ovr file with the same basename as the main filename, it was necessary to do something special so that overviews of subdataset would have one .ovr file per subdataset. This was accomplished by keeping the overview file name in the .aux.xml file associated with the subdataset.

As an added bonus it was decided that it was desirable to support building generic external tiff overviews for jpeg2000 images and to use those in place of the jpeg2000 derived overviews which can be fairly slow to access.

It turns out that .aux.xml metadata was supported for NITF subdatasets, and it was possible to build overviews for nitf subdatasets, and it was possible to substitute external tiff overviews for jpeg2000 data streams in an NITF file. But it was not possible to substitute external tiff overviews for jpeg2000 data stream in an nitf file with multiple images (subdatasets).

It took me a long time to figure out what aspects were broken, and how the various components were supposed to work even though I had implemented most of them. The problem is that many of these capabilities are rarely used, don't fit the standard GDAL model well, and were individually quite complex to implement. The complexity as the various aspects come together is compounded.

Each of the capabilities was added for fairly good reasons - mostly in order to provide a seamless, and performant user experience for GDAL users. But in order to provide this consistent external set of behaviors we are having to build more and more complexity into parts of GDAL - to the point where I am not sure it is sustainable.

Interestingly, most of this complexity has grown without input from the broader GDAL community. The PAM mechanism predated the modern Project Steering Committee and it's RFC process. I added the changes for PAM on subdatasets, and some of the specific NITF driver capabilities without discussion with the PSC on the assumption that they are either bug fixes, or are sufficiently driver-local that the PSC does not need to be involved. Possibly if these changes had needed to be justified in public, push back on the complexity might have prevented some.

The specific problem itself was fixed, as is documented in the ticket, with changeset 18312 holding the core fix. However, this fix is (IMHO) just adding additional fragility to the existing house of cards.

I don't really have a solution to the growing complexity, but perhaps thinking about, and starting to open up the issues a bit is a first step to containing the danger. There is certainly a cautionary tale or two in here.

BTW, GDAL 1.7.0Beta 1 is now released - testing and bug reports are welcome!

OGR DXF Driver

The last couple weeks I have been working on an OGR DXF driver. AutoCAD DXF format is a popular interchange format for CAD data, and to a somewhat lesser extent for goespatial map data often originating from engineering departments. It is a rather ugly format. Even though it is ASCII it is less than fun for humans to scan.

The raw machinery of the format is published by Autodesk, and lots of translators have been written for it in the past. However, I find it very frustrating that the format specifications fail to address the semantics of the format to any meaningful degree. It is assumed, I guess, that the person reading them is already deeply familiar with the AutoCAD data model.

So, for instance, it talks about the BLOCKS section, and the INSERT entity, but it never really explains that by defining a bunch of entities as a block, and then putting them into the drawing it makes it possible to treat a bunch of entities as a group. This has to be deduced. I have an old dxf file produced by FME that uses this as a mechanism to group a set of polylines that form a multipolygon, but I'm not sure how widely this approach is used.

In my driver, I have implemented two mechanisms for supporting blocks. In the case of non-text entities I try to accumulate the geometries I derive from the entities in the block into one OGC geometrycollection. But I also found that blocks in some drawings are used to group sets of text elements. In this case if I only use the geometries, I've discarded the important component - the text - so I collect the text entities as full features. And when I encounter the INSERT entity I push the original set of features into the input stream.

I spent most of a day implementing read support for the DIMENSION entity. This is essentially supposed to be a single object that shows a measurement on a drawing. The red part below is a single dimension element as rendered by QCAD.



Of course, there is no clear analog to this in OGR. So I ended up creating a single feature with a MULTILINESTRING geometry for the leaders, and arrows, and then pushed the label as a separate point feature with a LABEL style string.

I have made some effort to capture the drawing style information so that features can, in theory, be drawn similarly to AutoCAD by OGR applications. There is still really only one way to do this in OGR and that is to provide styling information as OGR Feature Style strings. This is a format that Daniel Morissette, myself and I believe Stephane Villeneuve defined many years ago as based closely on the sorts of styling supported by the mapinfo format(s). A few extensions have been made over the years, and a few OGR drivers utilize it to return styling information including DGN, and MapInfo. Andrey Kiselev has also done some stuff with it, so I presume there is a driver of his that uses it. So far the only two application that I know of using the feature style information are OpenEV and MapServer (via OGR autostyling).

I'm quite conflicted about the OGR feature style specification. I just hate to define "yet another" way of describing feature styling that is not closely related to any proper standard. I can't help but think there ought to be something more standards oriented to use as a basis. Perhaps OGC SLD, or SVG or something like that. But SLD didn't exist when we started, and does not express a lot of what I want. I don't really know that much about SVG - perhaps it would be a good choice. So, lacking a clear plan each time I need styling I do a bit more work on the OGR feature style specification while remaining less than fully committed to it as a long term thing.

Another aspect that was challenging was all the curves. I added the OGRGeometryFactory::approximateArcAngles() method as a generic mechanism to approximate arcs on an ellipsoid or circle as linestrings. The code was adapted from the Oracle driver and similar code exists in the DGN, and NTF drivers too I believe. So, finally this moves into the core.

One of the DXF curve types is a spline. Review the QCAD source code I found they implemented the spline rendering using rationale b-spline curves derived from Chapter 4 of An Introduction to NURBS by David Rogers. They release under the GPL so I can't directly use the code from QCAD without also putting GDAL/OGR under GPL restrictions, so I contacted the original author for permission to publish this code under the MIT/X license, like GDAL. He has indicated he has received my email and is considering the request ... so I wait.

In addition to the specification, I have also found the QCAD (and it's underlying dxflib library) to be a useful reference. I considered using dxflib but it really does not seem to solve any of the hard parts for me, and it would have added a dependency on an external library - complicating building of GDAL/OGR. The other very helpful resource was the v.in.dxf code from GRASS. This is a slightly less sophisticated dxf reader than OGR (IMHO) but it provided a very easy to understand implementation of a DXF reader for GIS data that was especially helpful in writing the proposal for the DXF driver. I also used it as a reference when implementing some of the element translations.

Before closing, I would like to thank Andreas Neumann and the City of Uster who have provided funding for this development. It seems that local government in Switzerland punches above it's weight in the free gis software world (the Kanton of Solothurn is also a big supporter of qgis, and related technologies as I understand). If only one in ten cities in the world made serious use of free gis software and provided enough financial support for one core developer it would have a huge boosting effect.

Anyways, preliminary DXF read support is in GDAL SVN now. Consider trying it out and providing feedbac.