Dataset Thinner for SLAM

Revised July 7, 2014, based on a code begun July 16, 2009.  
This code is copyright © 2010 by Vincent S. Cronin.  It may not be read, duplicated or used without his permission.

startTime=AbsoluteTime[];

Introduction

The purpose of this notebook is to minimize a large DEM dataset so that it can more efficiently be processed in SLAM.

Code

Test file

Change the style of the next line from Text to Input to test the subsequent code without having to use an external data file.  Make sure the code that is normally used to import a data file (in the gray box in the "Input the data file" section below) is disabled during the test -- disable it by converting the Import code line from an Input style to a Text style.

initialDataFile = {{"ncols", 6}, {"nrows", 7}, {"xllcorner", 0}, {"yllcorner", 0}, {"cellsize", 10}, {"NODATA_value", -9999}, {11, 12, 13, 14, 15, 16}, {21, 22, 23, 24, 25, 26}, {31, 32, 33, 34, 35, 36}, {41, 42, 43, 44, 45, 46}, {51, 52, 53, 54, 55, 56}, {61, 62, 63, 64, 65, 66}, {71, 72, 73, 74, 75, 76}};

Choose a reduction factor

A reduction factor (redFactor) of 2 deletes every other row and column;  the resulting matrix begins with the value in the first row, first column of the original matrix.  For example, if the initial matrix is a 7x6 matrix, the resulting matrix will be a 4x3 matrix with just under 29% of the values of the original matrix.  A reduction factor of 3 deletes two consecutive rows and two columns between values that are retained, resulting in a file that is ~14% of the original size.  A reduction factor of 5 deletes 5 consecutive rows and columns after each value it retains (~10% of the original size).

Assign an integer value to the reduction factor below.

redFactor=5;

Input the data file

This code reads an external data file that is in the form of an ASCII DEM file, with the .dat file-type designator.  On Vince Cronin's office Macintosh, the Import code would look like the following for a data file named "tahoedem.dat" located on the desktop:

initialDataFile = Import["/Users/vince/Desktop/tahoedem.dat"];

On the Dell computer in Cronin' s office, the Import code would look like the following for a data file named "tahoedem.dat" located on the C drive in the "TahoeRuns" directory :

SetDirectory["C:\TahoeRuns"];

initialDataFile = Import["tahoedem.dat"];

Use the appropriate Import code below, and make sure it is in the Input style.

initialDataFile = Import["/Users/vincecronin/Desktop/Trinidad/PurgatoireThirdDEM.dat"];

initialDataFile[[1,1]]

initialDataFile[[1,2]]

initialDataFile[[2,1]]

initialDataFile[[2,2]]

initialDataFile[[3,1]]

initialDataFile[[3,2]]

initialDataFile[[4,1]]

initialDataFile[[4,2]]

initialDataFile[[5,1]]

initialDataFile[[5,2]]

initialDataFile[[6,1]]

initialDataFile[[6,2]]

Extract the reduced data set from the original file

initialHeaderData=Table[initialDataFile[[i,j]],{i,6},{j,2}];

fatDEMdata=Drop[initialDataFile,6];

thinDEMdata=Take[fatDEMdata,{1,-1,redFactor},{1,-1,redFactor}];

matrixSize=Dimensions[thinDEMdata];

nrows=matrixSize[[1]];

ncols=matrixSize[[2]];

newCellSize=redFactor*initialHeaderData[[5,2]];

yllcorner=(initialHeaderData[[4,2]]+((initialHeaderData[[2,2]]-1)*initialHeaderData[[5,2]]))-((nrows-1)*newCellSize);

headerData={{initialHeaderData[[1,1]],ncols},{initialHeaderData[[2,1]],nrows},{initialHeaderData[[3,1]],initialHeaderData[[3,2]]},{initialHeaderData[[4,1]],yllcorner},{initialHeaderData[[5,1]],newCellSize},{initialHeaderData[[6,1]],initialHeaderData[[6,2]]}};

N[headerData]

mergedData=Join[headerData,thinDEMdata];

Export the resultant data file

Choose a name for the resultant data file (different from the name of the original data file) and specify the path to the directory where that file is to be saved.  For example, on Vince Cronin' s office Macintosh, the following code will save a resultant file named "thinTahoeDEM.dat" on the desktop :

Export["/Users/vince/Desktop/thinTahoeDEM.dat", mergedData];

on Vince Cronin' s Dell, the Export code would look like the following for a data file named "thinTahoeDEM.dat" located on the C drive in the "TahoeRuns" directory :

SetDirectory["C:\TahoeRuns"];

Export["thinTahoeDEM.dat", mergedData];

Use the appropriate Export code below, and make sure it is in the Input style

Export["/Users/vincecronin/Desktop/Trinidad/thinPurgatoireDEM.dat", mergedData];

How long did this program take to run, in minutes?

minutesForProcessing=(AbsoluteTime[]-startTime)/60