Spectra Artimetic Tool - Overlap 1D/2D spectra and calculate a relation (subtraction, addition, etc.)

This package provides a user interface and programming environment for conveniently aligning and comparing two 1D / 2D data waves and calculate a result from each overlapping data point:

  • result = f(modified input data, reference data)

Modes are ...

  • Subtraction (-)
  • Addition (+)
  • Multiplication (*)
  • Division (/)
  • Percent difference (difference normalized by the average in %, i.e., (A-B)/(A+B) * 200)
  • Custom user formula

The input data is modified via controls while a second data wave acts as the reference to compare against. The former can be shifted, offset and scaled manually with respect to the reference. The initial purpose of the package was just to calculate differences (hence the original name Difference Generator), but support for addition, multiplication, division, percent difference and later arbitrary formulas has been added.

The underlying original data of both the edited wave and reference wave is never modified in any way. The comparison is handled with temporary data, and modifications (default wave ending "_mod") as well as the arithmetic result are always saved as new output waves. The default wave ending for the output is "_dif" for subtraction, "_sum" for addition, "_mul" for multiplication, "_div" for division, "_pdif" for % difference and "_calc" for custom formulas. All settings are saved in the output wave's notes and will be reloaded automatically upon restarting the tool with the same input. The program is fully folder aware.

I tried my best to test everything thoroughly, but you should verify the correct behavior yourself to be sure. Bug reports and suggestions for new features are always welcome.

 

Graphical User Interface (UI) controls - 1D data

To start the user interface, select one or two (1D) waves in the Data Browser and then go to 'Spectra Tools' -> 'Compare and Modify Data....' in the top menu. Or use the command line:

WaveArithmeticUI(WaveToEdit)
or
WaveArithmeticUI(WaveToEdit,refwave=ReferenceWave)

The second wave selected in the Data Browser (or the wave provided by the optional 'refwave' parameter) comes pre-selected as reference when starting the UI. A quick overview of all controls can be found below:

The wave to modify and the reference wave are selected from the current folder. The current folder can be changed in the Data Browser during operation (while the UI is open) to use waves from a different folder. The list of waves in the drop-down menu will reflect always the available waves in the currently active folder.

  • If extend for full overlap is checked then the reference data is artificially extended (by repeating an average of the endpoints) to cover the full range of the modified wave, in cases where the waves do not fully overlap (if there is a full overlap already then nothing changes). Use this feature if you want to calculate the result over the full range of the input and the endpoints of the reference data are close to zero so that the error is small. This is only available of 1D data.

Arithmetic Mode:

  • Choose from the drop-down menu: M-R subtraction (- keyboard key), M+R addition (+ key), M*R multiplication (* key) or M/R division (/ key) or percent difference.
  • Alternatively, a custom equation can be entered using M (modified input data) and R (reference data). For example, entering (M+R)/2 will calculate the average of the input and reference.
  • The result can be inverted (flipped in Y direction) with the Invert checkbox (i key).

 Smoothing:

  • Both waves can be smoothed (Gaussian broadened). The smoothing is saved in the modified output (ending '_mod'). The smoothing of the reference wave is only for display and to calculate the result, but is not actually applied to the original reference data. However, the smoothing setting is saved with the output and will be reapplied upon reloading. Use the home / end keyboard shortcuts to increase / decrease smoothing (holding alt/option will smooth the reference data).

Applying Changes to the Input Data:

  • The data is shifted in X / horizontal and Y / vertical directions using the arrow buttons (cursor keys on the keyboard).
  • Scaling is done with the stretch (page up key) and shrink (page down key) controls.
  • The delta step of each modification can be set using the variable controls to the left.
  • Or use the controls for absolute changes to dial in the desired shift and scaling directly.
  • Use the reset buttons for all modifications or individual changes of the X/Y shift or scaling to undo the modification.

When you are done:

  • Discard & Quit will just close the interface without saving anything.
  • Save & Quit will save the current state as output waves and then closes the interface. Optionally, the result is displayed in a new graph afterwards if the relevant checkbox is active. At the same time, all previous graphs with the same data set displayed are updated as well to reflect the new modification settings. The state of all check boxes is saved with the output and will be reapplied upon reloading.

Additional keyboard shortcuts: 

  • Hold Command (Mac) / Ctrl (Win) for changes in steps of 0.5 times the current delta value.
  • Hold Shift for changes in steps of 10 times the current delta value.
  • Hold both Shift + Command / Ctrl for changes in steps of 0.1 times the current delta value.
  • r key to ‘Reset All'
  • p key to toggle ‘Pin Background’
  • Hold Command (Mac)/ Ctrl (Win) while pressing ‘Reset All’ and the reference wave will be set to none.
  • Hold Command (Mac)/ Ctrl (Win) while pressing ‘Save & Quit’ to save but without closing the tool. This can be used to compare many waves in quick succession, e.g., by selecting the next wave from the drop-down menu.
  • Hold Option (Mac) / Alt (Win) while pressing ‘Save & Quit’ to force 'Use Ref. Name' for saving (this has the same function as the respective checkbox).

 

Graphical UI controls - 2D data

2D data is loaded and processed in the same way as explained above. The main difference is that the data is now displayed as two overlapping images, with the reference data as a semi-transparent layer on top of the input data. There are a few additional controls specific to 2D data. First, there is now an additional control to modify the third (Z) dimension of the data:

At the bottom of the panel are controls to modify the image appearance:

  • Ref. Alpha: Controls the transparency (alpha value) of the reference. Choose between fully transparent (slider to the left) and fully visible (slider to the right).
  • Show (reference): Toggles the visibility of the reference image. Uncheck to hide. Can also be toggled with the 's' key on the keyboard.
  • Image color and Range controls: Choose the color mapping for each image and adjust the displayed z-range via sliders.

Showing Profiles and Setting Region-Of-Interest (ROI):

To help with overlapping the two data waves, intensity profiles in X and Y directions (the summed values the respective direction) can be displayed. Press the Show Profiles button and a side-panel will appear with two graphs, one for X and one for Y. Profiles will update automatically with each change, so you can leave the profile panel open to immediately see the effect of each change.

To limit the range of the data from which the profiles and min. / max. values are calculated, use the Set / Clear ROI button. First, drag a marquee across the image over the region of interest you want to work with, then press the button. Another press of the button without a marquee will clear the ROI again. You can alternatively use the right-click menu after dragging a marquee to set / clear the ROI there as well. This helps to match a specific region of the data.

 

How the result is saved

The result is saved next to the input wave (the data to edit) as 'wavename' +  "_mod"' for the modified data and  'wavename' +  mode-specific extension for the result of the calculation. The mode-specific extensions are:

  • "_dif"' (subtraction)
  • "_sum" (addition)
  • "_mul" (multiplication)
  • "_div" (division)
  • "_pdif" (percent difference)
  • "_calc" (custom formula)

All modification settings are saved inside the wave note of both of these output waves. Here is an example where 'first spectrum' was edited and 'second spectrum' was the reference:

Loading the tool with the same initial data ('first spectrum' in this case) will reload the last session from the '_mod' or '_dif' wave (if '_mod' does not exist). Note that loading a '_dif' or '_mod' wave directly will just start a new session with this data (this makes it possible to create differences of differences etc.). The input and reference data can be in different folders, but the '_mod' or '_dif' wave should stay with the respective input in the same folder.

If neither a 'dif' (or 'sum', 'mul', 'div') or 'mod' wave with the same name is found in the input wave's folder, then the default settings are loaded. It is thus recommended to leave the result and modified waves together with the input wave in the same folder to be able to reload the settings later. What will be reloaded if you have mixed result waves from different modes? The loading hierarchy is 'mod' > 'dif' > 'sum' > 'mul' > 'div' > 'pdif', e.g., deleting the 'mod' and 'dif' waves will load the settings from the 'sum' wave if present.

 

What is the purpose of the 'Use Ref. Name' setting?

Imagine the following scenario: A 'baseline' data and then several data sets have been measured in succession like this:

  1. baseline
  2. measurement1
  3. measurement2
  4. measurement3
  5. ...

Let us assume you want to remove the baseline wave from all the data sets, but the data itself should not be altered (for example, to preserve their relative shift or intensity). This is achieved by modifying (shifting, scaling etc.) the baseline for each measurement individually and then saving the calculation result next to each data set. Load the baseline as wave to modify and one measurement data set as reference, then do the modification (since the roles of edited wave and reference are reversed here, this usually means 'Invert' needs to be activated). Before saving check 'Use Ref. Name' to save the result next to each measurement data. This will give:

  1. baseline
  2. baseline_mod
  3. measurement1
  4. measurement1_dif
  5. measurement2
  6. measurement2_dif
  7. measurement3
  8. measurement3_dif
  9. ...

What if you reload measurement1 but have conflicting settings in waves measurement1_mod and measurement1_dif? In this case spectrum1_mod is loaded according to the hierarchy 'mod' > 'dif' > 'sum' > 'mul' > 'div' > 'pdif'.

 

Running your own code:

You can generate comparisons of two waves using the function: 

doWaveArithmetic(WaveArithmeticStruct s)

The WaveArithmeticStruct structure looks like this:

Structure WaveArithmeticStruct
    variable mode       // fixed mode of comparison: 1: Subtract (M-R) 2: Add (M+R) 3: Multiply (M*R) 4: Divide (M/R) 5: Percentage Diff => (optional; default is 1)
    char formula[100]   // free-form arithmetic expression containing M and R as stand-in for orgWave and refWave; use instead of mode
    Wave orgWave        // [mandatory] wave to compare 1 - will NOT be modified in any way
    Wave refWave        // [mandatory] wave to compare 2 - will NOT be modified in any way
    Wave cmpWave        // [optional] result of comparison between orgWave and refWave will be written here
    Wave modWave        // [optional] modified wave - saves the scaled and shifted version of orgWave
    Wave modRefWave     // [optional] modified reference wave - saves the smoothed version of refWave (if refSmooth > 0)
    // all parameters are optional
    variable orgSmooth  // amount of smoothing for wave 1           (default 0)
    variable refSmooth  // amount of smoothing for wave 2           (default 0)
    variable baselineXval   // x position of baseline               (default NaN)
    variable baselineYval   // y value (1D) or position (2D)of baseline     (default 0)
    variable baselineZval   // for 2D: z value of baseline              (default 0)
    variable baselineFixed  // baseline is omitted from Y scaling if set to 1   (default 0)
    variable invert     // invert the output in Y               (default 0)
    variable xShift     // modify wave 1: x displacement (in wave's units)  (default 0)
    variable yShift     // modify wave 1: y displacement (in wave's units)  (default 0)
    variable valShift   // modify wave 1: intesity displacement (offset)    (default 0)    
    variable xScale     // modify wave 1: x scale               (default 1)
    variable yScale     // modify wave 1: y scale               (default 1)
    variable valScale   // modify wave 1: y / z scaling             (default 1)
    variable UIpinToggle    // options related to user interface; don't use     (default 0)
    variable UIoptions  // bit options related to user interface; don't use (default 0)
EndStructure

In your code, use the structure to pass the input waves and desired parameters to the function. You need to pass at least the two input (orgWave and refWave) and one output wav (cmpWave; this one will be overwritten). A simple example code could look like this:

Function CompareTest(Wave wave1, Wave wave2)
    STRUCT WaveArithmeticStruct s
    WaveArithmeticInitialize(s) // initialize structure parameters with defaults
    Duplicate/O wave1, $(NameOfWave(wave1)+"_dif")
    Wave s.cmpWave = $(NameOfWave(wave1)+"_dif")
    Wave s.orgWave = wave1
    Wave s.refWave = wave2
    s.refSmooth = 0.5   // smooth wave2 (optional; data will not be altered)
    s.xshift = -0.02    // shift wave1 (optional; data will not be altered)
    String statusmsg = WaveArithmeticErrorMsg(doWaveArithmetic(s))
    Print statusmsg
End

Here WaveArithmeticErrorMsg() generates a readable error message, if there is a problem. If the same wave endings (e.g., "_dif") are used then the output is fully compatible with the user interface, i.e., the same settings will be loaded when starting the interface with the same input waves.

 

Settings inside the procedure header

The procedure header contains several settings as static constants which are used by the user interface, and can be edited to your liking (but you should not mess with these unless really necessary). For example, the name endings ("_dif", "_mod" etc.) for saving the result waves can be modified here, should there be a conflict with other names in your experiment (in fact, Igor's own Differentiate function will also use 'dif' as name ending):

static StrConstant kModEnd  = "_mod"                    // label for modified data
static StrConstant kAuxEnd  = "_smt;_int;_der;"             // label for additional data
static StrConstant kOutEnd  = "_calc;_dif;_sum;_mul;_div;_pdif;"        // label for output data
static StrConstant kModePresets = "M - R;M + R;M * R;M / R;(M-R)/(M+R) * 200;"  // presets for the modes

Project Details

Current Project Release

Release File: Difference Generator_v5.10.zip
Version: IGOR.8.00.x-5.10
Version Date: Tue, 03/05/2024 - 02:00 am
Version Major: 5
Version Patch Level: 10
OS Compatibility: Windows Mac-Intel
Release Notes:
  • Fixed bug: The 'Quit' panel had the wrong size when a graph expansion was active.
  • Fixed regression bug to work with older panel versions again.
  • Optimized a few code segments related to mouse dragging.
  • Removed minimum delta limit for y/z shifts and decreased the delta limit for x shifts, enabling more fine-grained shifts.
  • Fixed bug: Reference wave updates were skipped too often.
  • Fixed bug: Sessions were recreated from the first and not the last recording in case of multiple note entries.
View All Releases

Nice looking, chozo! It has a lot of features that aren't in the Wave Arithmetic panel.

Thank you John! As the name suggests, this project started to make it possible to create differences with manual control and instant feedback for extracting even minute differences between measurements. But now that you mention it, it has indeed some things in common with the Wave Arithmetic package. This actually gives me the idea to add division and multiplication to the package for completeness.

EDIT: I have now updated the project to include four modes: Difference, sum, multiplication, and division.

I am seeing a pattern here: it seems that spectroscopists keep toolbags of useful packages that do similar things. I have a procedure ("wave scale and subtract") that does some of the things your panel can do, but as usual your panel is more sophisticated and does a lot more.

I will attach my little procedure here, just in case you are curious, but it doesn't do anything other than a small subset of what you do here.

ScaleSub106.zip

At one point in the mid- to late 90's (Igor 3 to 4), the toolbox was collected under the package SpXZeigR. I had an entire suite of features, help files, and tutorials. You won't find the original anymore (I have it somewhere on an archived CD). You will find a scattering of references to it, including a doi article that I did not even know about until just now.

I always wished to have had the time (and/or co-developers) to sustain the package.

Yes, I remember SpXZeigR, and I had that and other packages in mind when I wrote the comment. I had already developed a set of tools more specific to the techniques I was using at that time, so SpXZeigR never really entered my workflow.

RE SpXZeigR ... Sad as I am to see the package fade into the historical records, I am also glad to see a plethora of well-designed packages being provided to take its place.

I remember SpXZeigR. If tech support queries reflects popularity, it was fairly popular :)

John: The support that WaveMetrics provided to sustain SpXZeigR was greatly appreciated.

I didn't mean that it had lots of problems! No matter how well designed your package is, people will get into trouble. So any package that gets used a lot will generate some number of queries, regardless of its quality.

Tony, thank you for sharing your solution for this problem. It looks very nice and sufficient for the task (I would miss the shift in x, though). I hope the DiffGen project will be of use for you (and others) then. Yes, the problems and solutions of people in the community seem to be similar. Good thing, that there is a place to share. I am happy that I didn't have to come up with the ~10 projects I am using from the site (mostly your stuff, Tony). :)

By the way, I didn't know or use SpXZeigR. Must have been before my time. ;)

Tech Note 20 still exists in all its variants. And it still works!!! But the tools you folks have been making available are far superior.

My packages supports menu keyboard shortcuts which are loaded from a settings file. If you want to add keyboard shortcuts to common menu entries, place the content (Spectra Tools settings.dat) of the attached zip file in the same folder as the package procedure and edit as necessary.

Spectra Tools settings_1.zip

Forum

Support

Gallery

Igor Pro 9

Learn More

Igor XOP Toolkit

Learn More

Igor NIDAQ Tools MX

Learn More