Binning data then taking stats on each bin

Share icon

I have 1D wave time series data (7000 points, for instance) that I would like to divide into bins (user-defined, let's say 100 points each, non-overlapping), and then I wanted to calculate the standard deviation in each bin. The sdev values would then form another 1D wave (of 70 points in this example), which I can plot and do with as waves go.

I know the WaveStats command can calculate the standard deviation (among other things), and one can also set the X range of the wave using /R=[startP, endP]. Having never programmed in Igor before, I'm not sure how to go about this programmatically, but I'm thinking of a placing the WaveStat command within a loop. Any help would be appreciated!
It appears that you want to create a Histogram (see the Igor help file). This makes a Histogram wave (with a default name, or name of your choice). Your example would have 100 points in the wave, defined by your bin choice. Then apply WaveStats to the Histogram wave.

Your description is a bit confusing. You cannot calculate a standard deviation for a bin, whose value consists of a single number. You can calculate statistics on the entire Histogram wave.

July 12, 2013 at 07:21 am - Permalink
I think compuchemgeek wants to find the standard deviations for 100-point segments of his wave. The term "bin" suggests histogram but that is not what he wants, I think.

Here is a solution:
Function StdDevBySegment(wIn, segmentSize, wOutName)
    Wave wIn
    Variable segmentSize
    String wOutName         // Name for output wave
    
    Variable numPoints = numpnts(wIn)
    Variable numSegments = trunc(numPoints/segmentSize)
    
    Make/O/D/N=(numSegments) $wOutName
    Wave standardDeviations = $wOutName
    
    Variable startPoint = 0, endPoint
    Variable segment
    for(segment=0; segment<numSegments; segment+=1)
        endPoint = startPoint + segmentSize - 1
        WaveStats /Q /M=2 /R=[startPoint,endPoint] wIn
        standardDeviations[segment] = V_sdev
        startPoint += segmentSize
    endfor
End
 
Function Demo()
    Make/O/D/N=7000 testData = gnoise(1)
    StdDevBySegment(testData, 100, "standardDeviations")
    Wave standardDeviations     // Created by StdDevBySegment
    DoWindow/F StdDevGraph
    if (V_flag == 0)
        Display/N=StdDevGraph standardDeviations
    endif
End


There may be a faster way to get just the standard deviation but I could not identify it.

July 12, 2013 at 09:53 am - Permalink
It may be easier to use the variance function directly since it supports the optional range parameters. An even more elegant approach is using MatrixOP (naturally). If your data wave is called "ddd" and you want N bins with M elements per bin you can execute:

Duplicate/O ddd,eee               // keep original data nice and safe.
Redimension/N=(M,N) eee     // every column corresponds to a bin
MatrixOP/O varData=VarCols(eee)


A.G.
WaveMetrics, Inc.

July 12, 2013 at 10:44 am - Permalink
Thanks to all who replied. I used A.G.'s elegant solution--the 3 lines of code did exactly what I wanted.

[quote=Igor]It may be easier to use the variance function directly since it supports the optional range parameters. An even more elegant approach is using MatrixOP (naturally). If your data wave is called "ddd" and you want N bins with M elements per bin you can execute:

Duplicate/O ddd,eee               // keep original data nice and safe.
Redimension/N=(M,N) eee     // every column corresponds to a bin
MatrixOP/O varData=VarCols(eee)


[/quote]

Regards,
Edward

July 17, 2013 at 09:05 am - Permalink