Lispix format

Is there any possibility to import raw-files in Lispix format into Igor:

https://www.nist.gov/services-resources/software/lispix
I would like to use Igor to handle "cube" data sets acquired from electron microscopy, which would be equivalent to 3D-waves, e.g. of the size "Make/B/U/N=(500, 500, 1024) cube". Those data are generated by spectral imaging, where each pixel of an image contains an energy dispersive spectrum.

Any ideas or suggestions how to import such data would be highly appreciated!

Cheers
Christian

The file format seems clearly documented.
GBLoadWave
is your friend.

You could first extract the parameters from the .rpl file using, e.g., Open, FReadLine and sscanf, and then use that information as input for the GBLoadWave operation. The nomenclature ("little-endian" etc.) is exactly the same in GBLoadWave and the documentation links you posted.

Wolfgang Harneit
... should go something like this (not tested! There's a little ambiguity in the doc about "little endian" being "low byte first"; also, scaling information is lacking in the official documentation)


#pragma rtGlobals=1		// Use modern global access method.

// adds a new menu to the right of the help menu
Menu "Lispix Support"
	"Import Lispix Files"
End

// to be called by user (e.g., through menu)
function/S ImportLispixFiles()
	variable refNum
	Open/D/R/T=".rpl"/M="Select one or more ripple (.rpl) files"/MULT=1 refNum
	string filePaths = S_fileName
	if( strlen(filePaths) == 0 )
		print "user cancelled."	
	else
		variable i, numFilesSelected = ItemsInList(filePaths, "\r")
		for( i = 0; i < numFilesSelected; i += 1 )
			string path = StringFromList(i, filePaths, "\r")
			ReadLispixFilePair(path)
		endfor
	endif
	return filePaths
end

// to be called by ImportLispixFiles()
function ReadLispixFilePair(ripplePath)
string ripplePath		// known to be a valid path to a ripple (.rpl) file

// 1. make sure there is an associated raw file, get its name
	string rawPath = ripplePath[0,strlen(ripplePath)-3] + "aw"
	GetFileFolderInfo/Q/Z rawPath
	if( V_Flag != 0 )
		print "no raw file found for "+ParseFilePath(0, ripplePath, ":", 1, 0)
	else
	
// 2. read ripple file, convert to keyword-value string
		variable refNum
		Open/R refNum as ripplePath
		string info = ""
		do
			string line
			FReadLine refNum, line
			if( strlen(line) == 0 )	// end of file
				break	// exit do-while loop
			endif
			if( !cmpstr(line[0],";") )	// comment line
				continue	// skip
			endif
			line = ReplaceString(" ", LowerStr(line), "")	// remove spaces and convert to lower case
			string keyStr, valStr
			sscanf line, "%s %s", keyStr, valStr
			info += keyStr + ":" + valStr + ";"	// convert to Igor-style keyword-value string
		while( 1 )	// exit is through break statement
		Close refNum

// 3. extract info from keyword-value string
		variable offset = NumberByKey("offset", info), byteOrder = 0
		if( !cmpstr(StringByKey("byte-order", info), "little-endian") )
			byteOrder = 1
		endif
		variable igorType = 0, length = NumberByKey("data-length", info)
		string dataType = StringByKey("data-type", info)
		strswitch(dataType)
			case "float":
				if( length == 4 || length == 8 )
					igorType = length/2
				endif
				break
			case "unsigned":
				igorType = 64
				// fall through
			case "signed":
				if( length == 1 || length == 2 || length == 4 )
					igorType += 8*length
				else
					igorType = 0
				endif
				break
		endswitch
		if( igorType == 0 )
			abort "illegal data type/length specification: "+dataType+"/"+num2str(length)
		endif
			
// 4. load raw file
		GBLoadWave /A=lispix /B=(byteOrder) /Q /S=(offset) /T={igorType,igorType} rawPath
		
// 5. report to history and rename wave to filename
		if( V_flag == 0 )
			printf "no waves"
		else
			printf S_waveNames
		endif
		print " loaded from "+S_path+S_fileName+"."
		if( V_Flag == 1 )	// should be one
			string oldName = StringFromList(0, S_waveNames)
			string newName = ParseFilePath(3, ripplePath, ":", 0, 0)
			wave oldWave = $oldName
			Duplicate/O oldWave, $newName
			KillWaves oldWave
			print oldName+" renamed to "+newName

// 6. redimension data cube
			variable width = NumberByKey("width", info)
			variable height = NumberByKey("height", info)
			variable depth = NumberByKey("depth", info)
			Redimension/N=(width, height, depth) $newName
			
// 7. set data scaling
			SetScale/P X, 0, 1, "Å", $newName
			SetScale/P Y, 0, 1, "Å", $newName
			if( !cmpstr(StringByKey("record-by", info), "vector") )
				variable Zscale = NumberByKey("ev-per-chan", info)
				SetScale/P Z, 0, Zscale, "eV", $newName
			else
				SetScale/P Z, 0, 1, "Å", $newName
			endif
	
		endif
	endif
end

harneit wrote: ... should go something like this (not tested! There's a little ambiguity in the doc about "little endian" being "low byte first"; also, scaling information is lacking in the official documentation)


Hi Wolfgang,

thanks for pointing me towards GBLoadWave! I feel quite guilty now, seeing that you wrote this code apparently between 2 and 4 a.m.!
Thanks for your effort - using this I got some test data imported straight away, although it looks different to what I expected - I will look more carefully at the parameters!

Thanks again,

Cheers
Christian
Do you get strange numbers like 6.5082e-317? then it's the byte-order logic and / or data format that is screwed up.
Wolfgang
the data range looks ok - it's more that the EDS spectra stored in the 1024 layers (I've expected to find counts per 1024 channels of the EDS detector), don't look like EDS spectra at all. Might be a problem of the data set, or the data need further conversion.... something I need to sort out.....

Cheers
Christian
The statement:

Redimension/N=(width, height, depth) $newName

assumes that the data in the file is stored in "column major order" (see http://en.wikipedia.org/wiki/Row-major_order).

For a matrix, this means that, as you sequentially step through memory, all of the elements for column 0 appear consecutively, then all of the elements for column 1, and so on. Column major order is used by Igor and, according to the wikipedia article, by Fortran and Matlab.

C and many programs use row major order. For a matrix this means that, as you sequentially step through memory, all of the elements for row 0 appear consecutively, then all of the elements for row 1, and so on.

I'm a bit unclear on how this translates to 3D.

If your data is in row major order then you will need to shuffle it to put it in the right order. You would do this by duplicating it to create a temporary wave and then using wave assignment statements to pick the data out of the temporary wave and put it in the output wave in the right order.

it seems that this is likely the problem, because the imported 3D wave shows some regular patterns such that rows, columns or layers might have been mixed up.
I will start with a 1D wave and try to figure out the order.

Thanks
Christian
On re-reading the doc ("...stored row by row (each row spectrum by spectrum)..."), it should probably be
Redimension/N=(depth,width,height) $newname

for vector cubes. Then, you would have images in the YZ dimensions, and the X dimension represents the spectra. I guess you would want XY as image dimensions and Z as the spectral dimension. This could be achieved by leaving the Redimension line as it is and then picking out the data with
newWave[][][] = oldWave[(p + q*width)*depth + r]

or so. Of course, in any case you'd have to modify the surrounding code a bit (e.g. defer the
KillWaves oldWave
statement, declare
Wave NewWave= $newName
, etc.)

Wolfgang

Last week I completed a hack to load raw WiRE files (proprietary filetype used in Renishaw Raman spectrometers, may hold 1d or 2d arrays of spectra and spatial coordinates). Jamie Boyd's BinaryReader procedure was indispensable for figuring out the structure of the binary files. I've used it to hack several file types, including Bruker FTIR files. If you haven't already tried it, you should check it out!

Edit: Oops, I just realized that this is an old thread that was resurrected when links were fixed. Still, I recommend BinaryReader to anyone reading this!