Batch curve fitting

I've made some progress with this and have updated my post. Sorry for all the edits, if anyone is reading!

I would really like some help with batch curve fitting. I have ~200 Y waves with corresponding X waves and would like to fit a user-defined function to each of these between two points. I need to save the coefficients for each wave to use for other things. Obviously, I would like to automate this.

I am using the Batch Curve Fit Procedure from the Help files (6.22A) which I've modified.

My user-defined function has 6 Coefficients. One is constrained and one is held for all fits, two others need to be the Y position at point 5 and 44 (start and end of fit), the other two I have a good guess for and these can float freely. How do I input the two that need to be read from each wave?

Any help would be very much appreciated.

I call the function using:
FitCRtolistofwaves(wavelist("MPD*", ";", ""),wavelist("Time*", ";", ""))


The user-defined function is elsewhere in the procedure window.

Function FitCRToListOfWaves(theYList,theXlist)
    String theYList
    String theXList
   
    Variable i=0
    string aWaveName = ""
    string bWaveName = ""
    Variable V_fitOptions = 4   // suppress progress window
    Variable V_FitError = 0 // prevent abort on error
    do
        aWaveName = StringFromList(i, theYList)
        bWavename = StringFromList(i, theXList)
        WAVE/Z aWave = $aWaveName
        WAVE/Z bWave = $bWaveName
        if (!WaveExists(aWave))
            break
        endif
       
        // /N suppresses screen updates during fitting
        // /Q suppresses history output during fitting
        Make/D/N=6/O W_coef
        W_coef[0] = {0,1,0.0006,0.0155,600,0.999}
        Make/O/T/N=2 T_Constraints
        T_Constraints[0] = {"K5 > 0","K5 < 1"}
        FuncFit/H="000010"/NTHR=0 /N/Q ChapmanRichards W_coef aWave[5,44] /X=bWave /D/R /C=T_Constraints
        WAVE W_coef

        // save the coefficients
        Duplicate/O W_coef $("cf_"+aWaveName)
        // save errors
        Duplicate/O W_sigma, $("sig_"+aWaveName)
        if (V_FitError != 0)
            // Mark the results as being bad
            WAVE w = $("cf_"+aWaveName)
            w = NaN
            WAVE w = $("sig_"+aWaveName)
            w = NaN
            WAVE w = $("fit_"+aWaveName)
            w = NaN
            WAVE w = $("Res_"+aWaveName)
            w = NaN
            V_FitError = 0
        endif
        i += 1
    while(1)
End


Any help would be much appreciated.
Hello sjr51,

I've actually been looking into your problem since this morning. Apologies for letting you think you're alone in the wilderness. I developed a Batch Curve Fit package for Igor 6.30, which is now in beta release, that solves a lot of your problems. You've got some tricky coefficient conditions, however, and it doesn't handle them all. I've been looking into what it would take to solve your problem, but meanwhile you've almost solved it yourself using the function that was the basis of the Batch Curve Fit package! Since you've almost got it yourself I'll first get you to the finish line.

You can set the coefficients you want to have the first and last Y values by setting them in the coefficient wave W_coef and then setting FuncFit's hold flag appropriately. So if coefficients 2 and 3 are to, respectively, take on the first and last Y values that code might look like the following:

Function FitCRToListOfWaves(theYList,theXlist)
    String theYList
    String theXList
 
    Variable i=0
    string aWaveName = ""
    string bWaveName = ""
    Variable V_fitOptions = 4   // suppress progress window
    Variable V_FitError = 0 // prevent abort on error
    do
        aWaveName = StringFromList(i, theYList)
        bWavename = StringFromList(i, theXList)
        WAVE/Z aWave = $aWaveName
        WAVE/Z bWave = $bWaveName
        if (!WaveExists(aWave))
            break
        endif
 
        // /N suppresses screen updates during fitting
        // /Q suppresses history output during fitting
        Make/D/N=6/O W_coef
        W_coef[0] = {0,1,aWave[5],aWave[44],600,0.999}          // Nate set the 2nd and 3rd index coefficient values to the first and last Y values
        Make/O/T/N=2 T_Constraints
        T_Constraints[0] = {"K5 > 0","K5 < 1"}
        FuncFit/H="001110"/NTHR=0 /N/Q ChapmanRichards W_coef aWave[5,44] /X=bWave /D/R /C=T_Constraints    // Nate held those coefficients
        WAVE W_coef
 
        // save the coefficients
        Duplicate/O W_coef $("cf_"+aWaveName)
        // save errors
        Duplicate/O W_sigma, $("sig_"+aWaveName)
        if (V_FitError != 0)
            // Mark the results as being bad
            WAVE w = $("cf_"+aWaveName)
            w = NaN
            WAVE w = $("sig_"+aWaveName)
            w = NaN
            WAVE w = $("fit_"+aWaveName)
            w = NaN
            WAVE w = $("Res_"+aWaveName)
            w = NaN
            V_FitError = 0
        endif
        i += 1
    while(1)
End


I changed 2 lines, commented with "// Nate ..."

That should solve your immediate problem. Let me know if it does not. I'll be able to look at it tomorrow morning, US Pacific time.


A cleaner, more generalized way to solve the problem would use an all-at-once fit function and make the first and last Y values a part of the fit equation but not one of the coefficients. You can't do this in a regular fit function because you don't have access to the full Y wave. For help on all-at-once fit functions type the following into your command line:

DisplayHelpTopic "All-At-Once Fitting Functions"

Note that inside the All-At-Once fit function the Y wave parameter has already excluded those parts of the range not included (e.g. 0-4 and 45+ in your case).

Finally, I note that there's a Batch Curve Fit package that I developed that ships with Igor 6.30. It makes a lot of the tasks setting up a batch curve fit easier, and includes a panel which visualizes results. Igor 6.30 is a free upgrade if you're using 6.something. See:

http://www.wavemetrics.net/Updaters/AboutPublicBetaVersions.html

6.30 is also a beta version, for which I'm grateful because I found a bug in the display portion of Batch Curve Fit when using it with an all-at-once fit function. I've fixed the bug but it will not be available until the next beta comes out. Thank you for finding a bug in the beta version without actually using it. Quite a remarkable feat! Also the Batch Curve Fit package does not include constraints, and its not easy to apply your first & last Y condition. I'm going to add constraints immediately, and the first and last Y condition would be doable using an all-at-once fit function.

Good Luck!

Nate
WaveMetrics
Nate Hyde
Dear Nate,

Thank you so much for your help! I have just tried your solution and it works really well: only one failed fit out of ~200 waves.

I will give the 6.3 beta release of Igor a try soon, although maybe I should wait for a release with the bug corrected.

Thanks again for your time,
sjr51
The bug applies only to viewing all-at-once fit results using the Batch Curve Fit panel, so if you're not using the BCF panel with an all-at-once fit 6.30 is fine as far as I know. I think your problem, however, is best solved with an all-at-once fit so yes, wait for the beta update.

If you're going to do more batch fits I think you'll find the package useful. For one thing it adds a nice panel for viewing results. The bug is fixed on my local version and I'll be done updating the UI to handle constraints in a day or two. If you update to 6.30 and send me an email at support@wavemetrics.com I'll send you the patched up Batch Curve Fit package ahead of the next 6.30 update. This offer goes for anybody else who reads this post. I hesitate to attach it here for fear that a user will grab the attachment BCF procedures at some future date when they are outdated.

Glad I could help!

Nate

Nate Hyde
WaveMetrics
Nate Hyde