Decimation of a data including gaps

Share icon

Fri, 02/15/2013 - 04:11 am

Hi all,

I have a dataset consists of more than thousend rows and several columns. Dataset has some missing points and I need to decimate this data by averaging every adjacent five data points. I want to end up with less data points. However there are several gaps in the data and when I try to use the following function, I end up with wrong numbers because they shift. Almost always I have five data points but sometimes I have only two and three missing points.

What I want to do is

- Apply the decimation by smoothing operation to this data which is attached as txt file.

My question is:

- Is it possible to find and insert the missing data points as NaN?

Decimated_F1W = mean(F1_W, x, x+4*deltax(F1_W))

I will be happy to hear your suggestions.

Cheers,
Emre 201209171810.Number_concentration.TSP_NaN.txt
tooprock wrote:
Is it possible to find and insert the missing data points as NaN?


If your missing data values are characterized by zeros, as it appears from your sample file, then this should work.

wave120 = wave120[p][q] == 0 ? NAN : wave120[p][q]

February 15, 2013 at 05:25 am - Permalink

No, they are not included to the sample file I attached. The NaNs that you see are the data points which I ignore. Missing points can be seen by looking at the date/time of measurement. Anyway, thanks for your message.

February 15, 2013 at 05:54 am - Permalink

You seem to indicate that there will be discontinuties in the time column, but it looks continuous to me. In the data that you posted there is a point every minute; I don't see where any points are missing. Can you post an example file with missing data?

If missing values are only found by examining the time values, then you will likely need to loop through the rows of your data looking for places where two adjacent time points differ by more than the correct step value.

My previous suggestion was based on the thought that it was the zeroes occuring in the field of data (not NANs) that constitued missing data. Sorry for my confusion.

Maybe it will be helpful to point out that when you find the gaps you can use InsertPoints to add a row of data which you can then set to NANs or whatever values you desire.

February 15, 2013 at 06:41 am - Permalink

Hello again,

You're absolutely right. There are discontinuties in the time column, which is actually not the case in the sample data file I sent. Imagine that the time scale is not homogeneous. How can I correct it? I thought that, I could write a loop and compare every two rows of time column. If the difference between tow adjacent value is larger than one minute (or 60 s in Igor) stop and calculate the difference and also add missing time values. If you gave me some hint, I would be grateful.

Best,
Emre

February 27, 2013 at 07:39 am - Permalink