VH File Formats

File formats read by VH Editor:

VH Editor can read a range of different file types, but with varying amounts of success.  This program can only deal with 1:24000 scale maps that have the coordinate data in UTM (Universal Transverse Mercator projection) format.  Technically, this program is also limited to UTM data that uses the "North American Datum of 1927" which uses the "Clarke 1866 Spheroid" to represent the Earth's surface.  If you read in data with a different spheroid model or with a different datum, the transformation from UTM coordinates to lat/long and back will be inaccurate (could be noticeably inaccurate).  This is on my list of things that need fixing!

Note that SDTS (Spatial Data Transfer Standard) is the extremely complex binary format that the USGS switched to in the 1990's for all map data except DRGs and ortho-quads.  I can only read SDTS files using a native C library compiled for the local platform and linked to the Java program using the Java Native Interface (JNI).  Currently, I only have this library compiled for MacOS X and Linux-i386.  If you have the ability to compile a C program into a dynamically linked library on your platform, I'd appreciate some help in this area.  See the SDTSj Library by following the Downloads link for more information.

DEMo 1:24k

Files with extensions:  *.demo, or *.dem

This is the old USGS ASCII DEM (Digital Elevation Map or Model) Optional format.  You can't get this format from the USGS anymore, but there are lots of quadrangles in this format floating around out there.  Also, there are converters out there that convert SDTS DEMs to this format.  Currently, VH Editor can only read DEMo files that have line feeds (the DEM standard does not require line feeds and so, some DEMs do not have them – all the data is on a single line).  If the DEM does not have line feeds, and if it has been padded to 1024 bytes per “line” (which is common), you can add line breaks every 1024 characters.  I have a simple C program that will do this if you need it.  Contact me for more information.

SDTS DEM 1:24k

Files with extension *.ddf

An SDTS transfer that happens to contain a DEM as a raster (columns and rows) of data.  I have never run into an SDTS DEM transfer that VH Editor couldn't read.  Recently (2001), the USGS discovered a bug in their program that converted all their old DEMo files to SDTS.  This bug introduced an error in the X & Y position of data elements.  So, they had to go back and reprocess all their old DEMs into the new format again.  So, if you have an old DEM SDTS transfer from before 2002, you need to throw it out and download a new one.  On the plus side, the newly created DEMs are of higher quality than the old ones with fewer data drop outs and systematic errors.

SDTS "data transfers" are made up of a large number of separate files with number/letter combinations for names that end in *.DDF.  All the files for a single data transfer must be in the same directory.  The names of the individual files are significant (but not meaningful to humans), so don't rename them.  When reading an SDTS transfer into VH Editor, you can select any file in the transfer, the program then determines what directory you are in and goes from there.

DLG 1:24k

Files with extension *.dlg

This is the old USGS ASCII DLG (Digital Line Graph) format. These easy to understand files are no longer available from the USGS.  However, there are a number of these files floating around on the net and they can be created by existing programs.  I am not aware of an SDTS to DLG converter that actually works 100% without a lot of hand reworking.  I have never run into a proper (not converted from SDTS) DLG file that this program couldn't read.

VH Editor does not support all node, line and area types that are possible.  In fact, it only supports a fairly small number of the possible attributes that you might run into.  When this program runs into an unknown type, it is drawn in a generic way (nodes as a black circle with a dot in the center, lines as a black line and unknown areas are not drawn).  If you are a Java programmer, it's very easy to add renderers for your own DLG types.  Contact me for more information.

SDTS DLG 1:24k

Files with extension *.ddf

An SDTS transfer that contains vector (Digital Line Graph -- DLG) data.  I have had great difficulty getting VH Editor to reliably read these files.  The problem is with attributes applied to the components (lines, nodes, areas).  In order to make the files smaller, they jumped through a a lot of hoops to have a whole set of cross-referenced lists that point from an element to one file to another to another that finally contains the attribute. By manually tracing it, I have often found errors in this cross-referencing scheme.  I can read about 80% of the SDTS DLG transfers that I've tried.  I especially have trouble reading hypsography layers (contour lines) because these layers break all the rules set up by other layer types when assigning attributes and I haven't figured out all their rules yet.

VH Editor does not support all node, line and area types that are possible.  In fact, it only supports a fairly small number of the possible attributes that you might run into.  When this program runs into an unknown type, it is drawn in a generic way (nodes as a black circle with a dot in the center, lines as a black line and unknown areas are not drawn).  If you are a Java programmer, it's very easy to add renderers for your own DLG types.  Contact me for more information.

SDTS "data transfers" are made up of a large number of separate files with number/letter combinations for names that end in *.DDF.  All the files for a single data transfer must be in the same directory.  The names of the individual files are significant (but not meaningful to humans), so don't rename them.  When reading an SDTS transfer into VH Editor, you can select any file in the transfer, the program then determines what directory you are in and goes from there.

GIF (1:24k DRG in GIF format with meta-data in external file)

Files with extension *.gif

A DRG (Digital Raster Graphic -- scanned high-res topo map) is just an image file along with some meta-data that identifies where the image is located in the world and what it's resolution is.  The standard format for DRG's is called GeoTIFF.  This is a TIFF image file that has the meta-data embedded in it.  Usually, the meta data is also provided in one of several different format external ASCII files.  Java can not read TIFF files directly (without a specialized library).  So, I usually convert the TIFF file to a GIF file that Java can read.  When you select a GIF file in VH Editor, it will look for the required meta-data in external files in the same directory with the same name (filename.fgd or filename.met) as the GIF file.  The meta-data file must be present to read in a DRG image.

TIz (Tiled Image Compressed format DRG with meta-data in external file)

Files with extension *.TIz

DRG image files tend to be very large (requiring 32-64 MB of RAM to hold one alone).  On my machine, I can not load more than two at a time.  As a result, I created a method in VH Editor that can read in the large DRG images (or any large image for that matter) and tile it into a number of small tile image files at various resolutions.  This allows you to effectively have dozens of DRG map layers open at a single time in Virtual Hiker without running out of RAM since it only reads in the tiles you need to display what you are currently seeing on the screen (plus a little more) at the current resolution.  When you select any TIz file in a set of TIz files in VH Editor, it will parse out the prefix used by all the tiles and it will look for the required meta-data in external files (prefix.fgd or prefix.met) in the same directory as the TIz files.  The meta-data files must be present to read in a tiled DRG.

Tech Note:  TIz files are just raw 32 bit image data in ARGB format, 512x512 pixels square, that has been gzip compressed.  This custom tiled image technique was developed a number of years ago, before Java Advanced Imaging (JAI) was available on my platform (MacOS X).  Now that JAI is available, this custom tiling approach may not be necessary.  But, on the other hand, it works well enough.  If someone would like to port the DRG renderer to use JAI to display regular DRG image files, that would be awesome!

MLST (Map Merge List file)

Files with extension *.mlst

Since it is possible for Virtual Hiker to read in dozens of map layers at a time, I quickly found that it was very inconvenient to actually merge in dozens of map layers one layer at a time every time I loaded the program.  So, I created an "index" file format called a "merge list".  You can think of a merge list as a simple list of layers to be merged into the same map window.  When you open the merge list with Virtual Hiker or VH Editor, it automatically opens all the map layers listed in it and merges them all together into a single map window (if that is possible).  It is a simple text file containing a list of partial paths to the map layers that you want to read in.  Look at an example of this very simple format in order to use it for yourself.

MAPz (Virtual Hiker native file format)

Files with extension *.mapz

This is the native binary file format used by Virtual Hiker and VH Editor.  It has the advantage of being able to store a collection of any number of map layers into a single file.  It is also compressed, easy for my program to work with, and loads very quickly.  However, it currently has the serious limitation that DRG map layers can not be saved to MAPz files.

Tech Note:  MAPz files are actually just a set of Serialized Java objects that have been gzip compressed.  This can cause annoying (though usually easily solved) problems with backward file compatibility as the program code changes.

Data Quality Issues

The data from the USGS often has serious errors. The DEMs often have data dropouts, or spikes (the pre-2002 "Mt. Rainier East" SDTS DEM has some dramatic spikes).  These I think I could probably deal with using automated processing techniques.  However, DEMs in areas of high terrain relief (mountains) also often contain systematic errors that result in "rows of ridges" in the data.  It looks like what you would get if you drag a large rake over the terrain.  Lots of furrows and ridges running parallel east-west.  I don't know how to fix this problem, and it appears to be fairly common.  It really shows up when  you extract contour lines from these DEMs too.

SDTS DLGs often have problems with their attributes.  Either elements do not exist at all, are not assigned an attribute (and so my program doesn't draw it) or they are assigned the wrong ones (and you get the wrong representation from my program).  While doing some quick comparisons of DLG quadrangles with topo maps, I've sometimes found literally dozens of errors.  Obviously the ability to edit DLG data needs to be improved in VH Editor.