Interpolate in different waves

Hi,all

I have 3 1D waves: 1 data/time wave, and 2 number waves. They are presumably all 5Hz data, but not strictly 5Hz data. Occasionally only 4 data per second or less. I now want to interpolate them all into full strict 5Hz data, is there such a method in Igor?

Cheers,

Mortoo

I recommend that you start experimenting with Analysis->Interpolate using linear interpolation.

For background information, execute:

DisplayHelpTopic "Interpolation"

If you need further help, I recommend that you attach a sample set of 3 waves here.

Hi Hrodstein,

Thank you for the reply. I tried the Interpolation. But it changed the rest of the data when interpolating. 

I attached 3 waves here, 1 time wave and 2 data wave.

I would like to know how can I interpolate the missing data with the neighbouring value. 

Cheers,

Mortoo

time wave (3.17 MB) data wave (3.17 MB) data wave (3.17 MB)

I examined your time wave. It is in fractional seconds. To see that, display it in a table and then choose Table->Format->Show Fractional Seconds.

I then created a table containing the time wave and the difference in seconds from each point in the time wave to the next point in the time wave. Looking at this in a table, I see something like:

2021-12-18 00:00:00.228 0.2039999961853027
2021-12-18 00:00:00.432 0.1959996223449707
2021-12-18 00:00:00.628 0.2030000686645508
2021-12-18 00:00:00.831 0.1970000267028809
2021-12-18 00:00:01.028 0.2030000686645508
2021-12-18 00:00:01.231 0.1970000267028809
2021-12-18 00:00:01.428 0.2039999961853027
2021-12-18 00:00:01.632 0.1960000991821289
2021-12-18 00:00:01.828 0.1999998092651367
2021-12-18 00:00:02.028 0.2030000686645508

Notice that the time differences (last column) are of unequal magnitude.

I then executed this to find the average time difference:

Print mean(timeDiffs), WaveMin(timeDiffs), WaveMax(timeDiffs)

This printed 0.208185  0  2.19483

I then did a histogram of timeDiffs:

So your times are roughly 5Hz on average but not consistently close to 5Hz.

You wrote:

I now want to interpolate them all into full strict 5Hz data

I did that with this command:

Interpolate2/T=1/I=3/N=(numOutputPoints)/Y=$dataOutName timeIn, dataIn

This command is from a function that I wrote called Create5HzData. It is in the attached experiment. It uses linear interpolation (/T=1) with "X From Dest" (/I=3). "X From Dest" allows precise control over the X locations at which the input data is sampled. The output is a waveform with the X data represented in the wave's X scaling.

Here are graphs, also from the attached experiment, showing the original data (data0 vs timeW - I renamed your waves for simplicity):

You can see that the output data, which is sampled every 0.2 seconds, clearly differs from the input data (note the bottom around 12:00 PM). This is because your data is very noisy. Therefore sampling every 0.2 seconds will miss the positive and negative peaks. Consequently the interpolated data is not a very good representation of your input data.

Although this does achieve your goal to "interpolate them all into full strict 5Hz data", the result is likely not what you are looking for. I think this problem will affect any kind of straightforward sampling.

Because of the nature of your data, I can't think of a practical way to achieve the goal. How to proceed depends on what you intend to do with the data.

 

 

 

 

Mortoo.pxp (19.27 MB)

In reply to by hrodstein

Hi Hrodstein,

Thank you for the help. Sorry for the misunderstanding. These 3 waves of data are collected directly from the instrument. So unequal magnitude is Okay for me. The "strict" I mean is just 5 data in 1 second. My goal is to interpolate the missing points by the neighbouring points. To make sure they are exactly 5 data per second, without changing other data. 

Cheers,

Mortoo

To make sure they are exactly 5 data per second, without changing other data.

I don't think there is a simple solution for this. I think it would require custom programming and even then I'm not sure it is feasible.

As a start to understanding the difficulty, think about the fact that each 1 second interval in your time wave is potentially different from the others in unpredictable ways. Try taking the first 10 seconds and manually deciding where you would insert a point in each 1 second interval.

 

In reply to by hrodstein

Hi Hrodstein,

How about creating a "strict" 5Hz time wave like

2021/12/18 00:00:00.00

2021/12/18 00:00:00:02

2021/12/18 00:00:00.04

......

and using a command to filter out data from the raw data within 0.2 seconds of the "strict" data to make a new wave. Unmatched data is displayed as NaN. 

Is there a command for this filtering function in Igor pro? Can "BinarySearch" command works here?

Cheers,

Mortoo

How about creating a "strict" 5Hz time wave like

This is what I did with interpolation. It would work with smooth data but not with spikey data, especially with very spikey like you have.

If you can come up with a fairly detailed algorithm to choose the data values for those times you have identified (every 0.2 seconds), then it could be programmed. I can not think of an algorithm that preserves the meaning of the data.

using a command to filter out data from the raw data within 0.2 seconds of the "strict" data

I don't understand that.

Is there a command for this filtering function in Igor pro?

I'm not sure what you mean by "filtering function". Some possibilities are Smooth, Extract, FilterFIR, and FilterIIR, but I don't see how any of them helps.

In reply to by zixuan.cheng

zixuan.cheng wrote:

and using a command to filter out data from the raw data within 0.2 seconds of the "strict" data to make a new wave. Unmatched data is displayed as NaN.

 

assuming monotonic time waves, this function selects the nearest point within plus-or-minus 0.1 seconds

function test(oldTimeWave, newTimeWave, oldDataWave, newDataWave)
    wave /D oldTimeWave, newTimeWave
    wave oldDataWave, newDataWave
       
    int i, imax, j, jmax   
    imax = numpnts(newTimeWave)
    jmax = numpnts(oldTimeWave)
    variable diff
   
    for (i=0,j=0;i<imax&&j<jmax;)
       
        diff = newTimeWave[i] - oldTimeWave[j]
           
        if (diff > 0.1)
            j++
            continue
        elseif (j<jmax-2 && abs(newTimeWave[i]-oldTimeWave[j+1])<abs(diff))
            j++
            continue
        endif
                   
        if (abs(diff) < 0.1)
            newDataWave[i] = oldDataWave[j]
            //j++
// incrementing j here prevents us from considering this point again, even though it may be a closer match for i+1
        else
            newDataWave[i] = NaN
        endif
        i++
    endfor
end

 

In reply to by hrodstein

Hi Hrodstein,

Sorry for the misunderstanding. I mean to create a new datawave with the same length as the "strict" 5Hz datetimewave. The data points of the new wave are selected from the most recent data from the old datawave.

Cheers,

Mortoo

In reply to by tony

Hi Tony,

Thanks for helping me again! I tried this function and it makes a wave with some gaps, but enough data points, which is exactly what I need. Thank you so much!


Another question is how to fill these gaps with neighbouring data. 

Cheers,

Mortoo

In reply to by zixuan.cheng

Hi Tony,

I replaced the "newDataWave[i] = NaN" by "newDataWave[i] = oldDataWave[j]" to fill the NaNs:

        if (abs(diff) < 0.1)
            newDataWave[i] = oldDataWave[j]
            j++
        else
            newDataWave[i] = oldDataWave[j]

Thank you so much.

Mortoo

In reply to by zixuan.cheng

For simply the closest point in time, I think this works, and fixes an error in the previous version:

function test(oldTimeWave, newTimeWave, oldDataWave, newDataWave)
    wave /D oldTimeWave, newTimeWave
    wave oldDataWave, newDataWave
       
    int i, imax, j, jmax  
    imax = numpnts(newTimeWave)
    jmax = numpnts(oldTimeWave)
   
    for (i=0,j=0;i<imax&&j<jmax;)
   
        if (j<jmax-2 && abs(newTimeWave[i]-oldTimeWave[j+1])<abs(newTimeWave[i] - oldTimeWave[j]))
            j++
            continue
        endif

        newDataWave[i] = oldDataWave[j]
        i++
    endfor
end