XPS data from Phy Multipak

Hi,

Has anyone have experience with .spe file from a Phy Multipak? I have a data file (attached) that I am trying to extract the data from. There should be 6 spectra that are defined in the header. Simpler files with only 1 data range I can sort of get, though there is a bit of garbage at the beginning of the wave after export.

edit: The exported CSV file is also attached.

My goal is to extract the data directly, because the export software does not support batch processing.

The multiple ranges a bit of a stump for me. Anyone with a magic .ipf that they can share?

Andy
C0ELR081_033.spe_.zip (4.04 KB) C0ELR081_HR_50_30.csv (11.27 KB)
Here is one way, starting from the csv file:

function FileLoader(Key, base, skip)       
    string key                  // keyword after which a spectrum begins
    string base                 // base name for spectra       
    variable skip               // line to skip after keyword
   
    variable refNum
    Open/D/R/F="*.csv"/M="Select SPE-CSV File" refNum
    string filePath = S_fileName
    Open/R refNum as filepath
   
    Make/FREE/O/T/N=0 w
   
    variable i =0      
    do
        string line
        FReadLine refNum, line
       
        if( strlen(line) == 0 )                         // end of file
            break                                       // exit do-while loop
        endif
       
        line = RemoveEnding(line, "\r")
               
        InsertPoints i, 1, w
       
        w[i] = line
        i+=1           
           
    while( 1 )
    Close refNum
   
   
    i=0
    variable  filenum = 0
       
    for(i=0; i<numpnts(w); i+=1)
       
        if(cmpstr(w[i], key) == 0)
            i+=skip
            Make/O/N=(numpnts(w)) $base + num2str(filenum)
            wave ww = $base + num2str(filenum)
            ww = NaN
        endif
       
        string ThisLine = w[i]
        variable strStart = strsearch(ThisLine, ",", 0)+1
        ww[i] = str2num(ThisLine[strStart, inf])
           
        if(strlen(w[i]) == 0)
            filenum +=1
            WaveTransform zapNaNs ww
        endif  
    endfor
end


try:
FileLoader("Area5", "data", 4)



The entire file is first loaded as text wave using FReadLine (maybe LoadWave would be easier), then I use a keyword (here "Area5") to figure out where a spectrum begins.
You need to add a bit of code to either extract x-data or to set the wave scaling.



Hi ChrLie,

Thank you very much for your code. I used it as a starting point and produced this using some patterns in the file. I would really love to be able to do this off of the .spe file. It has a text header with a lot of information needed including the number of spectra and the ranges to set the scaling. The actual data is in binary and I have not been successful in parsing it into the individual spectra.

Function ImportXPS()

    //get file
    variable refNum
    Open/D/R/F="*.csv"/M="Select SPE-CSV File" refNum
    string filePath = S_fileName
    Open/R refNum as filepath
 
    Make/O/T/N=0 w
    Make/O/N=0 SpectraStart
 
    variable i,j =0    
    do
        string line
        FReadLine refNum, line
 
        if( strlen(line) == 0 )                         // end of file
            break                                       // exit do-while loop
        endif
 
        line = RemoveEnding(line, "\r")
 
        InsertPoints i, 1, w
        //check to see if it is the start of a new spectra and note the line number
        if(stringmatch(line,"*Area*")==1)
            InsertPoints j, 1, SpectraStart
            SpectraStart[j] = i
            j +=1
        endif
        w[i] = line
        i+=1           
 
    while( 1 )
    Close refNum
   
    //make individual spectra
    variable SpectraIndex,index,offset
   
   
    For(SpectraIndex=0;SpectraIndex<numpnts(SpectraStart);SpectraIndex +=1)
   
        //Read in data
        Make/O/N=(0,2) DataIn
        offset = spectrastart[spectraindex]+4
        index=0
        Do
            if(strlen(w[(offset+index)])==0)
                break
            endif
            insertpoints index,1, Datain
            DataIn[index][0] = str2num(stringfromlist(0,w[(offset+index)],","))
            DataIn[index][1] = str2num(stringfromlist(1,w[(offset+index)],","))
            index +=1
//          print index
        while(1)
        //Make Spectra with Name
        Make/O/N=(dimsize(datain,0)) $w[spectraStart[spectraIndex]+2]
        Wave DataOut = $w[spectraStart[spectraIndex]+2]
        Dataout[]=Datain[p][1]
        //Set wave Scaling
        SetScale/I x, datain[0][0], datain[dimsize(datain,0)-1],  dataout
    EndFor
End
Andy,

Using a hex file editor, it isn't too difficult to figure out how to load the spectra. See the code below. The purpose of some elements of the file aren't clear (to me); however, the procedure loads these values into waves. The values are interpreted in ways that seem to make sense (i.e. as integers or floats) but I may have guessed incorrectly. Hopefully, they will make some sense to you. Also it wasn't clear how to get the wavelength information, but, maybe you already know how to do this. Oh, yes, this procedure will likely need modification if you want to load files with spectra from more than one position.

The procedure is embedded here and as a ipf file. Also attached is an experiment file with the results for your example file.

Hope this helps.

Jeff

#pragma rtGlobals=3     // Use modern global access method and strict wave access.

Function ImportXPS_bin()
//Read .spe files from Phy Multipak XPS
    Variable vFileRef
    Variable vIndex = 0
    Variable vLoc
    Variable vNumSpectra
    Variable vPointsInSpectrum
    Variable vReadVal
    String sFilePath
    String sLine
    String sSpectrumWaveName
   
//Get name/path to file to open
    Open/D/R/F="Multipak Files (*.spe):.spe;"/M="Select SPE-CSV File" vFileRef
    sFilePath = S_fileName
    if( StrLen( S_fileName ) == 0 )
        return -1   //user cancelled file selection
    endif
   
    Open/R vFileRef as sFilePath
   
//Text Header
    Make/O/T/N=0 wHeader
 
//Read the File Header one line at at time; read terminates on LF/CR combinations, per the help file
//In this case the terminator is LF
    Do
        FReadLine vFileRef, sLine
        FStatus vFileRef
        if( V_filePos > V_logEOF )
            print "Attempting to read past end of file"
            return -1   //attempting to read past end of file
        endif
 
        sLine = RemoveEnding( sLine, "\r" )
 
        InsertPoints vIndex, 1, wHeader
        wHeader[vIndex] = sLine    
    //get number of spectra in file from header containing
        if( stringmatch( sLine,"NoSpectralReg:*" ) == 1 )
            vLoc = strsearch( sLine, ":", 0 )
            vNumSpectra = str2num( sLine[vLoc + 1, inf ] )
            print "Number of Spectra in file: ", vNumSpectra
        endif

        //End of header is given by "EOFH" string; quit when this is found
        if( stringmatch( sLine,"EOFH" ) == 1 )
            print "End of header found"
            FStatus vFileRef
            print "Current file position: ", V_filePos
            break
        endif
        vIndex += 1        
    While( 1 )

//Binary headers
    //brief (16 bytes) header; seems to be 4 4 byte unsigned integer values, second valued may give number of spectra
    //otherwise the purpose of this part is not clear
    Make/O/N=4 wBinHeader_1
    //each spectrum gets 24 unsigned 4 byte integers; fifth value is number of points in corresponding spectrum
    Make/O/N=( 24 * vNumSpectra ) wSpectraHeader     
    //read the binary headers
    FBinRead/B=0/F=3/U vFileRef, wBinHeader_1
    FStatus vFileRef
    print "Current file position after bin header read: ", V_filePos
    FBinRead/B=0/F=3/U vFileRef, wSpectraHeader
    FStatus vFileRef
    print "Current file position after spectra header read: ", V_filePos

//Mystery value at end of each spectrum...
    //4 byte floating value at end of spectrum; don't know what it is for; read it anyway
    Make/O/N=( vNumSpectra ) wMysteryBytes 
   
//load each spectrum into a separate wave
    for( vIndex = 0; vIndex < vNumSpectra; vIndex += 1 )
    //name for spectrum wave; name is "wSpectrum_" plus number corresponding to position in file
        sprintf sSpectrumWaveName, "wSpectrum_%02d", vIndex + 1
        print sSpectrumWaveName
    //get number of points to load for the spectrum
        vPointsInSpectrum = wSpectraHeader[ 5 + vIndex * 24 ]
    //make the wave for the current spectrum
        Make/O/N=( vPointsInSpectrum ) $sSpectrumWaveName
        Wave wSpectrum = $sSpectrumWaveName
    //read spectrum; format is 64 bit float    
        FBinRead/B=0/F=5 vFileRef, wSpectrum
//deal with 4 bytes at end of a spectrum; either read into wave just for this or skip
//code for skipping is commented out
    //read 4 bytes at end of spectrum, interpret at 32 bit float
        FBinRead/B=0/F=4 vFileRef, vReadVal
        wMysteryBytes[ vIndex ] = vReadVal
//  //next 4 bytes have unknown purpose, skip for now
//      FStatus vFileRef
//      FSetPos vFileRef, V_filePos + 4
    endfor
   
    Close vFileRef
End
Phy_Multipak_SPE_Loader.ipf (3.41 KB) Phy_Multipak_Loader.pxp (49.23 KB)
How did you know how to interpret the *.spe binary file to get out the data?
I'm also working on a PHI machine and I'm also interested in directly loading the spe data.
McIlhenny wrote:
How did you know how to interpret the *.spe binary file to get out the data?
I'm also working on a PHI machine and I'm also interested in directly loading the spe data.


I used an application (a "hex editor", see the Wikipedia article.. http://en.wikipedia.org/wiki/Hex_editor) that permits viewing the contents of a file in binary form, bit by bit, if necessary. The text header in the file gave insights into interpreting the binary data and, certainly, having the text export version of the spe file was a big help. With those hints, you have to use the hex editor to examine the file for clues to its structure and make some guesses.

There was more information in the binary portion of the file that I did not understand. This was saved in a couple of other waves that, may make sense to someone with experience using the instrument. Hopefully, those mysterious portions were interpreted (as integer or floating point values) correctly.

Also, as I mentioned in my original post, it appears that spectra from different regions of the sample can be combined in the same data file. I don't think the code provided will handle this without some changes.

By the way, are you able to load your data with the Igor procedure that was posted?
Hi Jeff,

Thank you very much. I have never used a hex editor and this is certainly a good place to start. The spectra do make sense and match the ASCII values so your guesses were correct.

This great!

Thanks again.

Andy
Jeff,

Thanks for your positng.
Recently I strongly require data loader of the .SPE files.
The pxp or ipf file you uploaded gives an error "While sxecuting FBinRead, the following error occured: logical end-of file was reached unexpectedly uring read operation." when I execute 'importXPS_bin()".
I attach an example of SPE files.
Can you check it up? I am beginner of igor program.

JW
BP.118.zip (6.01 KB)
realMAX wrote:
Jeff,

Thanks for your positng.
Recently I strongly require data loader of the .SPE files.
The pxp or ipf file you uploaded gives an error "While sxecuting FBinRead, the following error occured: logical end-of file was reached unexpectedly uring read operation." when I execute 'importXPS_bin()".
I attach an example of SPE files.
Can you check it up? I am beginner of igor program.

JW


I will look into this.

Jeff
The 0.3beta version should load the spectra correctly.

FWIW, the mistake in the existing code was to read "mystery bytes" at the end of each spectrum.

Please test and confirm. I'll post a non-beta release based on feedback.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
Thanks a lot....jjweimer
I tested your IGOR.6.30.x-1.0 on both Igor versions 6.37 and 7.02.
As you noted, the error message always pops up.

In addition, there are several issues as the following.
The spe file that I uploaded consists of four spectra. But your macro only loads one of them.
I need a single wave with proper parameters such as row number, start, step per each spectrum. Total 7 waves are not necessary at all.
As you compare the Igor display with real spectrum in the attached figure, spectral shape and scale change a lot. Even the background is removed. I do not understand what happens...

Longing for better version,
realMAX
BP118.jpg (186.79 KB)
realMAX wrote:
Thanks a lot....jjweimer
I tested your IGOR.6.30.x-1.0 on both Igor versions 6.37 and 7.02.
As you noted, the error message always pops up.


Would you do me the favor and ...

* Log the error report to the Package (so that I can keep track of it there)
* Include a ZIP archive of your SPE file (so that I can troubleshoot on hand the data you have)

realMAX wrote:
The spe file that I uploaded consists of four spectra. But your macro only loads one of them.


I suspect this is because the SPE file that I tested only had one "region", whereas yours has more than one.

realMAX wrote:
I need a single wave with proper parameters such as row number, start, step per each spectrum. Total 7 waves are not necessary at all.


I prefer not to create wave formats that are different than what are currently used. The loader can always be run with a function that subsequently cleans up behind it or does other formatting/calculating steps, for example ...

Function MyLoadSPE()

    LoadPHISPE()
    setscale/I ...
    killwaves/Z ....
    return 0
end


As long as the data are all loaded correctly, decisions about what to do with the values and waves can be left to each user.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
jjweimer, Thanks

The error message is as follows
"While executing Make, the following error occurred: Expected name"
The zip file is attached again, the same file above BP_118. This is composed of 4 spectra.

Most serious problem is that the imported wave is different from real spectrum as I reported yesturday.

realMAX
BP.118_0.zip (6.01 KB) BP.118_0.zip (6.01 KB) BP.118_0.zip (6.01 KB)
OK. This took is a bit longer than I anticipated. I found the problem. Each "region" has five associated sub-regions.

SpectralRegDef: the XPS region definition
SpectralRegDef2: ??
SpectralRegDefBackgrond: ??
SpectralRegHero: ??
SpectralRegIR: ??

I will adjust the loader to skip past everything except the XPS regions.

--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH