Spectra Image Profile - Create profiles from, normalize and edit 2D / 3D spectral data

This project provides tools for working on 2D spectral data or layers of 3D data, and was inspired by the WaveMetrics Image Line Profile procedure (now replaced by the internal Line Profile tool). The basic functionality is similar to the official tool at first glance, but this project has many convenient features tailored for spectroscopic data such as detector images, two-dimensional scans or intensity maps. The main functionality is the horizontal or vertical data integration between two region-of-interest (ROI) lines to create a profile, which can be saved, compared or even used to modify the data itself. Other functions (such as normalization) are applied directly to the 2D data or layer. The tool works directly on the input data, but a backup is saved upon start. Several tools to work on 1D-3D data are included as well.

I tried my best to test everything thoroughly, but users should verify the correct behavior for critical applications. Bug reports and suggestions for new features are always welcome. Special thanks to Oleg Dyachok from Uppsala University for suggesting and testing improvements for version 4.0 and beyond.

User Interface controls

To start, select a 2D or 3D wave in the Data Browser and then go to ‘Spectra Tools’ -> ‘Image Profile …’ in the top menu. Or use the command line:

SpectraProfile(inputWave)

A quick overview of all controls can be found below:

Line profile controls

  • The profile can be calculated in horizontal (h key) or vertical (v key) direction. Diagonal or free-form modes (like in the Line Profile tool) are not available.
  • The Region-of-Interest (ROI) lines can be modified via the control variables, by dragging the lines directly with the mouse on the image or by using the cursor keys on the keyboard (holding the shift key speeds up movements by 4x and the ctrl / command key slows down movement by 1/2). Holding both the ctrl / command and shift keys slows the movement down to one delta unit (one row / column) for precise placement.
  • To instantly go to the full (minimal) ROI width use the respective check boxes or use the page up / page down keys.
  • Normalize Profile: The profile (not the image) will be normalized to fall between 0 and 1.
  • Mouse Csr Profile: An additional profile appears for comparison, which shows the current row or column under the mouse cursor (toggle with the m key). This profile is not saved.
  • Snap Lines to Grid: The ROI line positions are constrained to include only full rows / columns.

The Save Profile and Snapshot buttons

The Save profile button saves the current profile next to the input data using a name ending which depends on the selected range. The ending is of the format “_H(xxx-xxx)” (horizontal mode) or “_V(xxx-xxx)” (vertical mode), where xxx give the start and end of the current range, e.g., “mydata_H(16.7-19.3)”. If the full image is selected the ending will be just “_prH” or “_prV”, respectively. Holding the shift key while pressing the button will lock the ending to “_prH” / “_prV” regardless of the selected range. The range and width is always saved inside the output wave’s note.

The Snapshot button will instead create and display a backup of the current profile for comparison. Each button press updates this snapshot to the current one. This profile will not be saved.

Undoing changes to the 2D / 3D data

  • Undo All: Resets all changes made to the 2D/3D data and returns to the condition at start.
  • Undo Last: Reverts the last change made to the 2D/3D data by invoking a function. This will only work once and will not undo multiple steps.
  • Accept & Quit: Closes the window while keeping all edits made to the 2D/3D data so far.
  • Save as: Saves the 2D/3D data under a different name and undoes all edits made to the original data.

Normalization buttons

These functions work on the current layer. Currently, three normalization functions are implemented:

  • Div. by Profile: This divides the 2D image by the current profile (each column in horizontal mode and each row in vertical mode, respectively). The profile is slightly smoothed before the division to avoid amplifying noise. Holding the ctrl / command key while pressing the button prevents this smoothing step. See further below how to modify the smoothing parameter.
  • Each Maximum: Finds the maximum of each row (horizontal mode) or each column (vertical mode) and divides the row / column by this value. This is useful, for example, to equalize the data in one direction (use the Keep avg. Int. checkbox to prevent downscaling of the data to 1)
  • A user-defined button (here labeled ‘Ring Currrent’): Normalize by user provided information. This can be any data parameter or wave saved in the current experiment. Examples are normalization by a secondary readout, time, concentration etc. See the last section on how to define your own functionality here.
  • Keep average Int. checkbox: This will preserve the average intensity when normalizing the 2D data. Normalization involves division by some normalization term, which reduces the intensity. This checkbox will rescale the data back to the intensity level before normalization.

The Delete ROI Section button

This button removes the columns (horizontal) or rows (vertical) inside the Region Of Interest. If the data at the beginning of the wave is removed, the scaling is adjusted to account for the deleted points. However, if data in the center is removed the scaling cannot be adjusted, since a waves scaling must be always linear and continuous. Be aware that this may lead to a mismatch between the image data and the scaling. If the scaling is important (e.g., a continuous energy scale), either refrain from using this feature or take care to only remove sections at the edges. If 3D data is edited, this will delete the selected columns or rows on all layers.

The Subtract Profile button

The current profile will be subtracted from each column (horizontal mode) or row (vertical mode) of the 2D image or current layer of 3D data. This can be used, for example, to remove constant features from a measurement which appear undisturbed in part of the data and thus can be selected with the ROI lines. The Scale (x) control to the right of the button can be used to scale the intensity of the profile before subtraction.

This button has an additional functionality only in horizontal mode: The spectrum can be shifted each column step according to the Y-axis delta. In other words, each increasing column the Y-scale delta value is added to (or subtracted from) the profile’s x scaling, which subtracts the profile in diagonal direction. This is useful to subtract features which change with the parameter on the Y scale like, for example, spectral features moving with increasing photon energy.

  • Hold ctrl / command to subtract the profile along increasing Y delta values.
  • Hold alt / option to subtract the profile along decreasing Y delta values.

Below example shows the diagonal subtraction result:

The Shift +/- buttons

These two buttons only work horizontally (regardless of the selected mode) and are used to add or subtract the Y-axis scale from the X-axis. All layers of 3D data are processed. In other words, the X-scaling of each column is adjusted by its respective Y-scaling value; the data is recalculated into a parallelogram. This will straighten features in the data which were changing with increasing Y values. An example would be a photon-energy (Y) scan of a spectrum where certain spectral features increase in their (X) emission energy. Subtracting the photon energy scale from a photoelectron spectrum in a kinetic energy scale would give binding energy, while adding the photon energy scale reverts back to kinetic energy. The effect is illustrated again with above example data:

Hold ctrl / command to cut partial data after the shift has been applied (the parallelogram is cut back into a rectangle). Note that this might fail if there is not even one complete row of data left (the skew of the parallelogram is too large to fit a rectangle inside).

Align Features functionality

This algorithm looks for similar features of each column (horizontal mode) or row (vertical mode) of the 2D image or current layer of 3D data and tries to align these features for maximum overlap by shifting each column / row by a certain amount. The algorithm uses wave correlation and shifts the data in whole point steps (no interpolation) or sub-steps (interpolated) when the Sub-steps checkbox is enabled. This can be used, for example, to correct slight shifts of peaks in a spectrum or to estimate the shift of a peak versus some other parameter like time or position:

The algorithm uses the first column (bottom to top in horizontal mode) or row (left to right in vertical mode) of the data as reference and tries to align all other columns / rows to this reference. Hold shift to use the last row / column instead (top to bottom in horizontal mode and right to left in vertical mode).

Often, it is not desired to align to the full data with lots of different features. Setting the A and/or B cursors onto the image will limit the range from which the reference is selected to maximize overlap with only one selected feature. This might still fail if many similar features are present.

  • Activate Smooth to smooth the data before aligning. This can improve the results for noisy data.
  • Activate Align Edges to search for maximum overlap of edge features instead of peak features. The derivative of the data is used for this purpose.
  • Substeps is activated by default, which aligns features in steps smaller than the distance of data points, i.e., by using interpolation. Deactivating this feature will instead move data in integer steps.
  • Some data points will be outside the input range when shifting data. Usually, these out-of-bounds data points are set to NaN. Activating Fill Gaps will instead repeat the end points of the data to fill this region.
  • Activate Save Result to write the shift values into a wave with the name ending "_shft". The output is scaled by the 2D data’s wave scaling.

Note that the algorithm is ‘dumb’ in a sense that it only looks for the maximum correlation and not if peak features really match etc. In the worst case this just aligns artifacts such as noise.

Image contrast and color controls

You can choose the used color table and adjust the contrast of the image or currently selected layer by using the drop-down menu and the adjustment sliders. Select the desired color table and slide the Min and Max controls until the contrast is as desired. The sliders only have a range between the maximum and minimum data value of the image or currently selected layer, so the contrast cannot be reduced further, only increased. If you reverse the position of the Min and Max sliders then the colors of the image will be inverted. Check the log scale checkbox to switch to a logarithmic scale for both the image and profile. Switching to this mode will not work if the data only contains negative or zero values.

 

Additional controls for 3D data (volumes)

Layer and Merge controls

The Layer setting chooses the currently displayed layer of a 3D data set. The profile will only be created from this layer and most functions for editing data will only act on this one layer (there are exceptions like Delete ROI Section and Shift). The Delete button will delete the currently shown layer. The Merge All button will merge all (remaining) layers into a single 2D image which can then be worked on further. This can be reverted by using the Undo buttons. The scaled position of the current layer will be displayed in the top-right corner of the image.

Displaying profiles from all layers

Activating Multiple Profiles will show profiles of all layers, which are generated from the current ROI, simultaneously for comparison with the main profile from the current layer (shown in a bold line). This is useful, for example, to follow small changes in signal throughout the layers. The Separation control can be used to change the vertical spacing between the profiles. This spacing is in percent of the maximum y value. These profiles will also be saved in a wave with the ending "_map". The maximum number of profiles (layers) for this functionality is 100.

Clicking Display Z-Profile will open a small graph which plots the change of intensity in z- (i.e., layer-) direction. This can be useful to track variations across layers. The type of information displayed can be changed via the pop-up menu. Currently average, maximum or minimum intensity can be selected.

Rotating a 3D volume

3D data can be rotated by 90 degrees around the principal axes x,y,z in clockwise and counter-clockwise direction. Use the designated buttons at the bottom of the panel for this purpose. Rotations can be undone using the Undo buttons. After rotation the top-most layer is displayed at the beginning. If the layer is empty (only zeros) then the image might look all white. This is no bug, but the normal way how empty layers are displayed. Rotation might reverse the x- and/or y-axes, depending on the new orientation of the volume's contents. For this reason, the data might 'look reversed' when displayed in a normal image plot.

 

Quick access functions in the menu

Some useful functions are available in the Spectra Tools menu:

  • Append 1D,2D Datasets into 2D Data: Combines the columns (Y dimension) of multiple datasets (e.g., multiple measurements) into one consecutive 2D data wave. The X dimension is extended to cover the full X range of all datasets if needed.
  • Stack 2D Datasets into 3D Data: Stacks multiple 2D spectra (e.g., MCP images) on top of each other to produce a 3D array.
  • Profile of 2D\3D Data: Same as creating a profile over the full data in horizontal or vertical mode using the profile panel. This will give 1D data for a 2D input, or 2D data or a 3D input. Hold shift while invoking the menu to define a sub-range in the respective dimension for profile generation.
  • 2D Image from 3D Data: Sums 3D data in the layer (array) direction to give a 2D image (e.g., gives a sum of all MCP images of a 3D array).
  • Split Up 2D\3D Data: Separates 2D / 3D data into individual waves holding columns, rows or layers. The individual waves are saved in a folder with the name of the input.

In Igor 9, these options are also available in the right-click menu of the data browser.

 

Creating profiles, splitting or combining data from the command line

Some quick access functions can be directly invoked. To create a profile use:

QProfile(input_wave, dimension [,from ,to ,from_Point ,to_Point])

Here dimension is an integer value from 0 to 2, with the meaning of 0 = vertical profile (average rows), 1 = horizontal profile (average columns), 2 = z profile (average layers). Use the optional 'from', 'to' or the 'from_Point', 'to_Point' values to define a sub-range in terms of scaled values or point values, respectively. If both types are specified the former takes precedence.

Data can be split along a dimension via:

QImageSplit(input_wave, dimension)

Here, the dimension parameter has the same meaning as above.

To reduce the fidelity of data you can bin data points by using:

QuickBinData(input_wave, binX, binY, binZ)

Here, binX, binY, binZ should be integer values greater or equal to 1, which decide by how much the data in the respective dimension is binned. For example, using binX = 2 will result in a wave with half of the points in the x (row) dimension. Data can be binned in multiple dimensions simultaneously.

To combine data in the x- (rows), y- (column) or z- (layer) directions, use:

QuickCombineData(dimension, wave_list)

The dimension parameter has the same meaning as above. wave_list is a string list of waves (use full paths if the waves are not in the current folder).

 

Procedure settings and user button code

The procedure header has two global constants which control the smoothing factor of …

  • kNormProfileSmooth: Sets the smoothing factor as an integer for profile normalization. The profile is smoothed by this factor before used for division with the 2D data.
  • kAlignSmooth: The initial smoothing factor applied to each row / column before the correlation is calculated. Useful to suppress noise. The factor can be adjusted at any time inside the panel.

The user-defined (normalization) button

Two functions and one global variable are used to provide the functionality for the user button:

  • kUserInfoBtnName: Defines the button’s label and can be changed to reflect the purpose of the button. This name should be short to fit inside the button.
  • Profile_ActivateUserInfoButton(inputWave): A function for checking whether the conditions for using the button are fulfilled. A return value of 1 activates, any other value disables the button. One could for example check here if the required information is contained in the 2D data or loaded in the current experiment.
  • Profile_ExecuteUserInfoButton(inputWave): This function is called to do the normalization (or whatever you like to do here). Return a scaling factor for scaling the intensity to its average after the normalization step should the Keep average Int. checkbox be activated (returning 0 skips this step).

You can modify the constant and the two functions directly at the beginning of the procedure file, but it is more convenient to provide your own code in a separate file or inside the experiment environment. Use the Override keyword in front of the Function / StrConstant statement to override the standard definitions inside the package. Have a look at the example implementation inside the project’s procedure, which is used to normalize the ring current of synchrotron experiments saved inside the 2D data itself.

While the user button is intended for normalization, it is really a free-for-all button for any modification to the 2D data you may want. You could instead write a function which, for example, splits the image in half or applies an interpolation or does something completely different in each experiment file.

The following code shows the current implementation of the user-definable normalization button:

Static StrConstant kUserInfoBtnName = "Ring Current"    // the title of the user button

Static Function Profile_ActivateUserInfoButton(inwave)
    Wave inwave
    Variable activate = 0
    // +++ conditions for activation +++
    activate = StringMatch(GetDimLabel(inwave,1,0),"*mA*")  // check if dim-labels contain "mA" values
    activate = StringMatch(note(inwave),"*ring current normalized*")? 0 : activate  // already normalized?
    // +++
    return activate // return 1 to activate button
End

Static Function Profile_ExecuteUserInfoButton(inwave)
    Wave inwave
    Variable valAvg = 0
    // +++ ring current normalization procedure +++
    Variable i, value
    for (i = 0; i < DimSize(inwave,1); i += 1)
        String Currlabel = GetDimLabel(inwave,1,i)  // get the information from the column label
        sscanf Currlabel, "%s (%f mA)", Currlabel, value    // search for ring current info in the form "time (XXX mA)"
        if (value != 0)
            inwave[][i] /= value
        endif
        valAvg += value // add the values for intensity re-normalization later
    endfor
    valAvg /= i+1
    Note inwave, "ring current normalized"  // write into note that the normalization has been done
    // +++
    return valAvg   // return the intensity normalization factor
End

 

Project Details

Current Project Release

Release File: Image Profile_v4.40.zip (33.26 KB)
Version: IGOR.8.00.x-4.40
Version Date:
Version Major: 4
Version Patch Level: 40
OS Compatibility: Windows Mac-Intel
Release Notes:
  • Added toggle to fill gaps created by the align feature.
  • Added scaling to the profile subtraction feature.
  • Fixed a crash with Igor 8.
View All Releases

Wow. That single handily replaces 90% of the scripts I use every day with one convenient panel!

Thank you for sharing ?

Great to hear. Thank you for your comment! I think this tool saves me the most time as well (others are graph-related tools which I haven't put up here yet ;). I guess the other 10 % are specialized scripts only related to your field? Please let me know if there is a feature which would make your (and others) life easier.

I couldn't help noticing the images for the Subtract Profile and Shift +/- sections. Those images look very similar to the Auger-yield NEXAFS data that we were battling with for a long time. In the end we used an iterative process to split up the data into vertical NECAFS and diagonal XPS contributions. If you are working with the same problems you might be interested in the paper.

"Removing photoemission features from Auger-yield NEXAFS spectra" https://doi.org/10.1016/j.elspec.2017.05.012

Thank you for your comment and the link to the paper. I will take a look. Yes, this was exactly the reason for this functionality. The Subtract function relies on the fact that the XPS signal is present undisturbed in part of the scan and thus can be subtracted from the mixed image. We achieved this by simply measuring a sufficiently large range below the edge in question. This is of course complicated by second-order signal contributions, which are however often sufficiently small in our spectra. It might be interesting to add your approach to this package. What do you think?

I did notice you expanded the kinetic energies far enough to where the Auger peaks are gone. I guess there are still some secondary electrons left, but it's good enough to where you can extract a reasonable XPS spectrum and subtract it.

If you want to try my approach out you can find it here www.wavemetrics.com/project/EccentricXPS

There is a pretty good help file and an example .pxp file you can test the cleanup procedure on. If you think it would fit into your package I would be happy to help you add it, but it's probably best if you test it first and see if you think it will fit.

My packages supports menu keyboard shortcuts which are loaded from a settings file. If you want to add keyboard shortcuts to common menu entries, place the content (Spectra Tools settings.dat) of the attached zip file in the same folder as the package procedure and edit as necessary.

Spectra Tools settings_0.zip (468 bytes)

This is great and thank you! Question, is it all possible to think about adding a Mode to visualize the 2D amplitudes across the layers in a 3D stack as they evolve with the Z-axis? Perhaps we could have it extract based on cursor location, or a specified Xmin-Xmax: Ymin-Ymax range that users change? I have data sets that display intensities over two frequencies (X & Y) that then evolve over time (Z). So, I have codes that extract that data automatically from specified ROI or from every single point in the 3D waves. But this generates large amounts of waves and graphs just to visualize the data trends. Igor also has good extraction tools, but to my knowledge this still requires extracting and plotting to visualize the data. Overall, images/maps are great for visualizing peak changes with frequency and/or time - no doubt. However, I spend a lot of time analyzing intensity changes at specific points, looking for rate differences, oscillations, FFT residuals, and then restacking said residuals into images. 

Do you think adding a mode to visualize intensities plotted against the Z axis at specific X-Y coordinates is a feasible addition?! =)

Hi Electron, Thank you for your message and feature suggestion. This sounds quite interesting. I think it would be possible to show the evolution of intensity as a function of z at the mouse location and/or integrated overt the region of interest between the red lines. I don't think it will be that easy to show the intensity based on an user-chosen sub-region (like a marquee), since there is no code to handle marquees yet. Would a display at the mouse location enough for you? Would you want to know the integrated or 2D variation (z vs. x or z.vs. y, which would be a 2D image)?

In reply to by chozo

Hi Chozo, thanks for your feedback. Displaying from the mouse location and/or integrated over a region would be very useful. I think showing the 2D variation over X or Y would be interesting additions. However, I initially had envisioned showing the intensity at the given point a specified 1D wave. I think even the option to just show the Z intensity against number of points (layers) as one moved the cursor around would be very helpful! I hope that makes sense, and I am happy to assist in anyway I can - although my programming skills are still young =).

OK, attached is a start. You have to activate both 'Display z-Profile' and 'Mouse Profile' for now, since I have implemented only this one feature. Then you can move the mouse across any point to see the profile there. Regarding the integrated z-Profile, I am still unsure how to display this information, i.e., either as (fake) waterfall plot or image. What do you think?

Spectra Image Profile v231110.zip (29.57 KB)

Nevermind, it was not so difficult to add a function to integrate the ROI in both x- and y- to create a z profile. Please try out the attached procedure and let me know if that is what you have envisioned.

Spectra Image Profile v231110b.zip (29.85 KB)

@Electron: I have now added this feature into the official version, although in a slightly different way as in the test version.

Hi! This is a great code for spectra image analysis, many thanks! I have quite a silly question since I am trying to plot the peak shift vs energy for some of my data. The 2D data I have does not contain the energy scale and I typically make the image plot for my 2D data vs energy (1D wave). However, in the image profile of the Spectra tools kit, it only plots the 2D wave, thus I am wondering if there is a way to be able to choose the x scale of the plot so that the peak shift will correspond to energy shift...

Great to hear that you like the package! The support of scale-waves is on my long term feature list, but it turns out this interferes with the (red) profile lines especially for non-linear x waves.

I am not sure I understand fully what you are trying to do. What do you mean by "peak shift"? Is this about the output of the 'Align' feature? You could simply use the output and plot it versus your 1D wave for scaling. I think you would need to recalculate the scale a bit, since x waves for 1D and 2D data works a bit differently.

Would it be possible to attach some example data here? Then I can guide you through the process.

I appoligize, it might be confusing and hopefully will be more clear with some data. I try to send in here a 2D data (spectra as a function of time) and 1 D data (energy scale) as an example. I am interested into plotting the peak position in energy (or shift) as a function of time for the different spectra recorded. As for now the output of the 'Align' feature gives me some shift (in some numbers which are not connected to energy) vs time. I would like the y-axis to be energy of some kind. Hope that this became a bit more clear. Thank you!

test_Mika.pxp (1.82 MB) test_M.png (1.51 MB)

No need to apologize. I see what you want to try clearly know. First note that the Align feature compares and shifts whole rows (unless you specify a limited range via cursors). The output is in data points, i.e., how many points each row has been shifted over to match. So you would need to multiply these number with the energy delta, i.e., how much energy each point corresponds to. Looking at your energy scale, there is quite some fluctuation; the step between each energy is not constant. What you can do is to assign an 'average' energy delta to the output. This is the same as applying a linear scale to your image before using my Image Tool. Here is how you can do that. Execute the following for your example data:

Duplicate/O '2D_wave_shft', '2D_wave_shft_en'
'2D_wave_shft_en' = '2D_wave_shft'[p] * (energy_scale[numpnts(energy_scale)-1]-energy_scale[0])/numpnts(energy_scale)

The last factor is basically the average energy step calculated from energy_scale.

If you want to properly take the variable energy spacing into account then you would need to first interpolate your 2D_wave into a linear scale. You could use interp2D for that, but it is a bit more involved. Then you use my Image Tool on the transformed data. Since your energy-step fluctuations are quite small compared to the rather large shift you observe, the result would not be that affected (i.e., the error you get with above simple approach is small). Let me know if you are interested and I can walk you through that as well.

Many thanks, Chozo! This seems to work pretty well now so it is already good for my data! Since I have plenty of data with even smaller energy shift it might be worth looking into the linear scale interpolation of the image if it is not too much to ask for (just to make sure I don't miss any shifts?). Will it be somehow possible to obtain the energy position (instead of shift) of the peak vs time? 

The Align feature does not know anything about your data, it's shape, position etc. So it is impossible to define a 'position' automatically for arbitrary data. You can do the following, however: Find out the peak position at the beginning yourself and then just add the energy shift to that value. I fitted the larger peak at the very bottom of your 2D image and got 96.976 points for the start position, which is:

print energy_scale(96.976)
  18.1231 eV

Now you can do:

'2D_wave_shft_en' = 18.1231 + '2D_wave_shft'[p]*(energy_scale[numpnts(energy_scale)-1]-energy_scale[0])/numpnts(energy_scale)

This gives you the position for that 'peak'. Note that the shape is not well defined at the beginning of 2D_wave. You could also use some position which is well defined at the center of the data and work from there. Just make sure you find the correct offset. Note that you can also use the Align feature in reverse: Hold 'shift' to align to the top row as reference. This might be better if the bottom rows are noisy or have some other issue. Maybe at some point in the future I offer to align to an arbitrary reference.

I will get back to you regarding the interpolation later.

In reply to by chozo

Thank you very much, Chozo for all your help! This really saved me some time! I have checked now some other data with smaller shifts and I think the approach you suggested works well enough for my data. So don't bother about the interpolation, this is already very good and I am very thankful. 

Great to hear that. Just let me know when the interpolation might become necessary or you have any other requests.

Forum

Support

Gallery

Igor Pro 9

Learn More

Igor XOP Toolkit

Learn More

Igor NIDAQ Tools MX

Learn More