How Interactive Can I Make Graphs and Functions?

My inquiry requires a bit of background, so bear with me.

Background: I map out impact craters on other planetary surfaces.  I've used Igor since around 2007 to analyze crater rims by doing basic circle fits to crater rims.  I do the actual mapping in GIS (Geographic Information System) software (ArcMap) and export the data in simple lists of latitude and longitude that is then read into Igor for analysis.

I've upgraded last year writing a very naïve AI detection code that nonetheless has sped up my workflow by a factor of ~5x.  The AI code – originally written in Igor but converted to Python because it was faster – returns a list of what it thinks are craters, returning the lat/lon center and approximate size.  I wrote further code in Igor that takes in the images, presents me with the location of each feature (one at a time), and lets me grade the feature as either a true positive or a false positive.

Once that's done, I then need to identify the craters it missed.  My current workflow is to take the user-curated AI list back into ArcMap, create simple points in the middle of craters that I think the AI could find if I tell it where they are and a very rough size (I have three shapefiles for different size ranges), or do what I've done for 15 years and trace out crater rims for those features that an AI won't be able to do well.  I then take those point data and the rims back into Igor to do the analysis.

What I'd Like to Do: Anyone who's worked with ArcMap knows it's a real pain.  I'd like to move away from it.  I'd really like to be able to at least do the point identification in Igor and have it find my craters.  Ideally, in Igor:

  • I would have a graph window that shows my image or a portion of my image.
  • Overlaid would be all the craters already identified (simple scaled markers that are circles).
  • Igor would be on "standby" waiting for me to do something.
  • I could create a simple point in a crater center, or line, or 3 points by just clicking on the image, somehow.
  • As soon as that is done, Igor would run a set of functions I've written that takes that input and runs the crater detection in that immediate location, finding the crater, and drawing it on that image.
  • It would present a dialog box where I can enter a simple 1 or 0 (yes/no) that it did it right.
  • If it did it right, then it saves that feature and pauses for me to do the next one.
  • If it did it wrong, it erases and lets me try again, and I would either do it slightly differently (if I had done a point, maybe do 3 points this time) or just skip it and I'll do the full rim in ArcMap.

I fully understand Igor was not designed for this level of interactivity.  I have written a very kludgey way to do this where I can use the Show Tools tool on a graph and make a 3-point polygon, then run a function or macro from the command line to analyze it.  However, because of the constant window switching and manually running things each time, it is much slower for me to do it in Igor than the more complicated back-and-forth of my current Igor-ArcMap method.  So, I think the only way to make this of comparable or faster speed is to have that sort of interactivity where Igor is effectively waiting for that user input on the graph window and knows when to execute an analysis function upon completion of the user feature annotation.

For what it's worth, I have looked at the documentation for PauseForUser, but I have not explored it given the warnings about stability and that it seemed a bit complicated and like it might not be what I want.  I'm also running Igor 8; if there's an Igor 9 feature that makes this possible, I'll upgrade.

PauseForUser is probably what you want to use in this situation. You've probably done this already, but make sure you read this section:

DisplayHelpTopic "Pause For User"

I don't think you're proposing anything that's particularly difficult to implement in Igor, though you have a rather complex workflow that is performance sensitive so you may need to put a decent amount of effort into this to get it working exactly how you want.

Even though I'm not sure that IP9 will be significantly better than IP8 for what you're doing, I do suggest that if you're going to spend a significant amount of your time on this that you use the latest version. IP9 does have quite a few performance improvements that could be useful to you. It's also maintained, so if you discover a bug (not unlikely with PauseForUser) it's something we'll possibly fix in the version you're using.

Regarding this one point:

  • It would present a dialog box where I can enter a simple 1 or 0 (yes/no) that it did it right.

Consider using DoAlert with alertType=1. That will give you a true modal dialog with Yes and No buttons. You can press the Y key (or enter) to select Yes and the N key for No.

BTW, I should mention the ImageAnalyzeParticles operation to you in case you haven't already looked into that. Also, if you need help making your code run faster, let us know (best through emailing your experiment and/or code to support@wavemetrics.com).

Look at windows hook functions with the SetWindow command that will allow you to run a function just by clicking your mouse in an image without having to move the clunky cursors around. Maybe you could even click and drag to indicate the size of the crater using the mousedown and mouseup event codes.

For showing the location of already-identified craters you could use DrawArc which will allow you to draw circles using the coordinate system of the image. Another option if your images are black and white is to expand them to have red, green and blue channels and give the already identified craters a slight color.

aclight is right that this will take some effort to get right, but if you have another 15 years in you, then it's well worth it!

The main window might be the graph containing the image and a control panel subwindow with perhaps three buttons: Done, Start New Crater, Analyze. You might or might not want Erase Current Crater. Use a window hook to field mouse clicks; the window hook would presumably maintain an XY wave pair to display the clicks.

The Analyze button would do its thing, then put the result on the graph, and as aclight suggests, use DoAlert to ask if the result is good.

Actually, perhaps you don't need the Start New Crater button- that would simply be the starting state, and the Analyze button would simply do the analysis, wait for your answer, store the new crater or not, reinitialize and cycle around to waiting on the window hook again.

My recollection is that a while back I "helped" you with this project, and perhaps I suggested ImageAnalyzeParticles and it wasn't really appropriate. Correct me if I'm wrong! My fading recollection is that the craters tend to have only partial rims, so they don't really make good particles.

I'd use GraphWaveDraw/GraphWaveEdit to convert mouse click positions to wave entries. And continue from there.

> I fully understand Igor was not designed for this level of interactivity.

Designed or not, you can get anything interactively done.

Shameless plug: See https://youtu.be/Y3I-LDX-R2Q?t=465 for a demo of a program we are developing in Igor Pro. The dude explaining the program is me ;)

As you can tell, in Igor there's usually more than one way to get a task done! We pride ourselves on the programmability of Igor, including the control panels that allow "sophisticated UI programming". I think that's what's in the data sheet...

It all sounds very doable to me.

My inclination would be to create an interface with a graph window and a panel. I would try to avoid pauseforuser as much as possible and use a hook function to capture mouse clicks. You can also hook keyboard events if you want to build a very efficient interface.

Consider: mousedown, define and draw a circle centred at mouse position, set a flag for active drawing; mousemoved, redraw circle so that mouse position is on rim; mouseup, reset flag.

You could plot points as markers and allow the markers to be dragged for refining their positions. GraphWaveEdit can be helpful here: the UI for adjusting the points is provided, but comes with limitations. For an example of using GraphWaveEdit for interactively adjusting point positions, take a look at the spline drawing part of the baselines project.

I have also played around with using GraphWaveDraw to allow a user to draw a region of interest. The trick there was to create a panel with continue/cancel buttons, set a window hook to catch mouseup at end of GraphWaveDraw and bring panel to front, start GraphWaveDraw, then set pauseforuser and wait for the panel to be killed, then remove window hook.

I would say that Igor absolutely is designed for that level of interactivity, and the time investment for getting up to speed with UI design is not too bad compared with other options.

Here is an example I played with. It's just meant as an idea of what you can do.

If you run the MyCraters() function, it creates a fake image of some craters. In the left image where some of the craters are already colored, you can color more craters by clicking with the mouse anywhere in the image. Move the mouse and click again to indicate the size of the circle to be colored. The size of the circle will be updated as you move the mouse around in the image.

Updating the image on the fly as you move the mouse is a little dangerous because it can get very slow, but it seems to work OK here.

Function MyCraters()

    //  Creates a wave for making fake craters
    Make/O/N=2000 root:FakeCraterRadialWave/WAVE=FakeCraterRadialWave
    MultiThread FakeCraterRadialWave[0,149]=1e-4*(x-50)^2-0.75
    MultiThread FakeCraterRadialWave[150,199]=0.4e-07*(x-200)^4
    MultiThread FakeCraterRadialWave[200,*]=0
    //Display/K=1 FakeCraterRadialWave
   
    //  Creates the image wave with craters in it
    Make/O/N=(1000,1000) root:CraterImageWave/WAVE=CraterImageWave
    FastOP CraterImageWave=0

    //  Defines fake x,y positions and sizes of five craters to be created in the image
    Make/O root:CraterPositionWave={{400,200,1},{300,700,1},{200,400,0.4},{850,300,1.7},{550,800,0.5}}
    Wave CraterPositionWave=root:CraterPositionWave
    MatrixTranspose CraterPositionWave
    //Edit CraterPositionWave
   
    //  Adds the craters to the image
    Variable NumberOfCraters=DimSize(CraterPositionWave, 0)
    Variable i=0, x0=0, y0=0, size=0, x1=0, x2=0, y1=0, y2=0
    for (i=0; i<NumberOfCraters; i+=1)

        //  Finds the x,y position and relative size of the crater
        x0=CraterPositionWave[i][0]
        y0=CraterPositionWave[i][1]
        size=CraterPositionWave[i][2]

        //  Finds the affected area in the image to update
        x1=Max(x0-200*size, 0)
        x2=Min(x0+200*size, DimSize(CraterImageWave, 0)-1)
        y1=Max(y0-200*size, 0)
        y2=Min(y0+200*size, DimSize(CraterImageWave, 1)-1)

        //  Adds rhe crater
        MultiThread CraterImageWave[x1, x2][y1, y2]+=FakeCraterRadialWave( 1/size*Sqrt((x-x0)^2+(y-y0)^2) )
    endfor
   
    //  Spinkles magic moon dust on the image to make it look more "real"
    Duplicate/O/FREE CraterImageWave MagicMoonDustWave
    MultiThread MagicMoonDustWave=gnoise(0.5)
    Smooth/DIM=0 100, MagicMoonDustWave
    Smooth/DIM=1 100, MagicMoonDustWave
    MultiThread MagicMoonDustWave+=gnoise(0.1)
    Smooth/DIM=0 10, MagicMoonDustWave
    Smooth/DIM=1 10, MagicMoonDustWave
    MultiThread MagicMoonDustWave+=gnoise(0.1)
    FastOP CraterImageWave=CraterImageWave+MagicMoonDustWave

    //  Creates the color version of the crater image and scales it between 0 and 65535
    Make/O/N=(DimSize(CraterImageWave, 0), DimSize(CraterImageWave, 1), 3) root:CraterRGBImageWave/WAVE=CraterRGBImageWave
    CraterRGBImageWave[][][]=CraterImageWave[p][q]
    FastOP CraterRGBImageWave=CraterRGBImageWave-(WaveMin(CraterRGBImageWave))
    FastOP CraterRGBImageWave=((2^16-1)/WaveMax(CraterRGBImageWave))*CraterRGBImageWave

    //  Brightens and darkens the individual channels in the RGB image to add a color cast to the craters
    Variable Radius=0, Color=0
    for (i=0; i<NumberOfCraters-1; i+=1)
        x0=CraterPositionWave[i][0]
        y0=CraterPositionWave[i][1]
        Radius=CraterPositionWave[i][2]*150
        ColorCraters(CraterRGBImageWave, x0, y0, Radius, Color)
    endfor

    //  Plots the black and white image
    DoWindow/K CraterWin
    Display/K=1/N=CraterWin/W=(100,100,100+500,100+500);AppendImage CraterImageWave
   
    //  Plots the color image
    DoWindow/K CraterRGBWin
    Display/K=1/N=CraterRGBWin/W=(700,100,700+500,100+500);AppendImage CraterRGBImageWave

    //  Associates a hook function with the window, allowing craters to be marked with a click of the mouse
    SetWindow CraterRGBWin hook(MarkCrater)=MarkCraterHook, userdata(CraterCenter)=""
   
    //  Creates a copy of the RGB image which will be used when coloring craters
    Duplicate/O CraterRGBImageWave, root:CraterRGBImageWaveCopy
end



Function ColorCraters(CraterRGBImageWave, x0, y0, Radius, Color)
//  Brightens and darkens the individual channels in the RGB image to add a color cast to the craters
Wave CraterRGBImageWave
Variable x0, y0, Radius, Color

    //  Finds the affected area in the image to update
    Variable x1=Max(x0-Radius, 0)
    Variable x2=Min(x0+Radius, DimSize(CraterRGBImageWave, 0)-1)
    Variable y1=Max(y0-Radius, 0)
    Variable y2=Min(y0+Radius, DimSize(CraterRGBImageWave, 1)-1)
   
    //  Calculates the radial distance from the center to each point in the image as r=sqrt((x-x0)^2+(y-y0)^2)
    //  This way of doing it looks complicated, but is a lot faster
    Duplicate/O/FREE/R=[x1, x2][0][0] CraterRGBImageWave, XPosWave, XMultWave
    Duplicate/O/FREE/R=[0][y1, y2][0] CraterRGBImageWave, YPosWave, YMultWave
    MultiThread XPosWave=x
    MultiThread YPosWave=y
    FastOP XPosWave=XPosWave-(x0)
    FastOP YPosWave=YPosWave-(y0)
    MultiThread XPosWave=XPosWave^2
    MultiThread YPosWave=YPosWave^2
    FastOP XMultWave=1
    FastOP YMultWave=1
    Duplicate/O/R=[x1, x2][y1, y2][0] CraterRGBImageWave, RadialPosWave
    MatrixOP/O/S/NTHR=0 RadialPosWave=Sqrt((XPosWave x YMultWave) + (XMultWave x YPosWave))

    //  Updates the colors
    switch(Color)
        case 0:
            MultiThread CraterRGBImageWave[x1, x2][y1, y2][0]=( RadialPosWave(x)(y) <= Radius ? (65535-(65535-CraterRGBImageWave[p][q][1])*0.8) : CraterRGBImageWave[p][q][r])      //  Makes the red channel brighter
            MultiThread CraterRGBImageWave[x1, x2][y1, y2][2]=( RadialPosWave(x)(y) <= Radius ? CraterRGBImageWave[p][q][1]*0.8 : CraterRGBImageWave[p][q][r])                  //  Makes the blue channel darker
            break
        case 1:
            Wave CraterRGBImageWaveCopy=root:CraterRGBImageWaveCopy
            Duplicate/O CraterRGBImageWaveCopy CraterRGBImageWave
            MultiThread CraterRGBImageWave[x1, x2][y1, y2][2]=( RadialPosWave(x)(y) <= Radius ? (65535-(65535-CraterRGBImageWave[p][q][1])*0.8) : CraterRGBImageWave[p][q][r])      //  Makes the blue channel brighter
            MultiThread CraterRGBImageWave[x1, x2][y1, y2][0]=( RadialPosWave(x)(y) <= Radius ? CraterRGBImageWave[p][q][1]*0.8 : CraterRGBImageWave[p][q][r])                  //  Makes the red channel darker
            break
    endswitch
end



Function MarkCraterHook(s)
//  The hook function used for marking craters
STRUCT WMWinHookStruct &s
Variable hookResult=0
String CraterPositionString=""
Variable x0, y0, Radius

    //  Mousedown: s.eventCode=3, Left click: s.eventMod=1
    if ((s.eventCode==3) && (s.eventMod==1))
   
        //  Indicates that an action has taken place
        hookResult=1
       
        //  Reads the userdata of the window. I could have used a global variable instead, but this way the information stays with the window and is deleted with the window
        CraterPositionString=GetUserData(s.winName, "", "CraterCenter" )
       
        //  Checks if this is the first click defining the position of the crater
        if (StrLen(CraterPositionString)==0)
       
            //  Updates the userdata with the new center position
            SetWindow $s.winName userdata(CraterCenter)=Num2Str(AxisValFromPixel(s.winName, "bottom", s.mouseLoc.h))+";"+Num2Str(AxisValFromPixel(s.winName, "left", s.mouseLoc.v))+";"

        //  This is the 2nd click defining the size of the crater
        else
       
            //  The position and size of the new crater
            x0=Str2Num(StringFromList(0, CraterPositionString, ";"))
            y0=Str2Num(StringFromList(1, CraterPositionString, ";"))
            Radius=Sqrt((x0-AxisValFromPixel(s.winName, "bottom", s.mouseLoc.h))^2+(y0-AxisValFromPixel(s.winName, "left", s.mouseLoc.v))^2)
           
            //  Updates the colors of the displayed crater image
            //  This is calculated while the mouse is being moved, so the calculation has to be pretty fast!!
            Wave CraterRGBImageWave=root:CraterRGBImageWave
            ColorCraters(CraterRGBImageWave, x0, y0, Radius, 1)

            //  Makes the new coloring permanant
            Duplicate/O root:CraterRGBImageWave root:CraterRGBImageWaveCopy

            //  Clears the crater position from the userdata
            SetWindow $s.winName userdata(CraterCenter)=""
        endif
           
    //  Mouse moved
    elseif (s.eventCode==4)

        //  Reads the userdata of the window
        CraterPositionString=GetUserData(s.winName, "", "CraterCenter" )

        if (StrLen(CraterPositionString)>0)
       
            //  The position and size of the new crater
            x0=Str2Num(StringFromList(0, CraterPositionString, ";"))
            y0=Str2Num(StringFromList(1, CraterPositionString, ";"))
            Radius=Sqrt((x0-AxisValFromPixel(s.winName, "bottom", s.mouseLoc.h))^2+(y0-AxisValFromPixel(s.winName, "left", s.mouseLoc.v))^2)
           
            //  Updates the colors of the displayed crater image
            //  This is calculated while the mouse is being moved, so the calculation has to be pretty fast!!
            Wave CraterRGBImageWave=root:CraterRGBImageWave
            ColorCraters(CraterRGBImageWave, x0, y0, Radius, 1)

            //  Indicates that an action has taken place
            hookResult=1
        endif
    endif
    return hookResult
end

 

Wow! Thanks for all the suggestions.  It'll take me awhile to wade through them, though it's nice to see that there are a couple different options.  Other than it actually working, my main issue is it being fast.  If it's not faster than my current workflow, I won't use it, since I'm doing literally hundreds of thousands of these so a half-second extra per crater propagates to over a hundred hours extra time in the long-run.

As an example of a simple UI thing to speed stuff up, for the "UserDecides()" function I wrote where I grade the AI-based craters as true or false positives, it's pre-seeded with a "1" so I can just press the Enter key if it's a real feature; otherwise, type 0 and press Enter.  One less keystroke saves time!  And my AI-based initial pass is good enough that about 91–93% are true positives, so that's a lot fewer "1"s to type.

Technically, you shouldn't need the "enter". You could set it up so you only need to press "0" or "1" and automatically advance to the next crater.

How does your workflow work? Do you have a very large image with a slightly oversensitive AI so you are sure all craters are identified. Then all you have to do is weed out the 7-9% false positives? How does that work, do you get a small cropped image around each crater you cycle through and hit "0" or "1" for each image?

In reply to by olelytken

olelytken wrote:

Technically, you shouldn't need the "enter". You could set it up so you only need to press "0" or "1" and automatically advance to the next crater.

How does your workflow work? Do you have a very large image with a slightly oversensitive AI so you are sure all craters are identified. Then all you have to do is weed out the 7-9% false positives? How does that work, do you get a small cropped image around each crater you cycle through and hit "0" or "1" for each image?

I also allow a "-1" to go back if I went too quickly, so waiting for a single keystroke might not be best.

The AI is under-sensitive since I'd rather it get low-hanging fruit easily than struggle to get harder stuff and have a lot more false positives.  If I can speed up half of my crater mapping by 5x, then that's still a lot of savings.  The images I'm using are enormous mosaics (~260 GB in the equatorial region of Mars, ~340 GB for mid latitudes, each covering about 3% of the planet) that I've chopped into 600 parts with small overlaps that are each around 250 MB.  There is a bit of lag in Igor drawing/zooming to each new area because each snip is so large, but it's maybe half a second so noticeable but not horrible, and I have it also display the "upcoming" crater so my eye can already make a decision on that one while the UI is updating the main window.  Main feature in the attachment is on the left, upcoming on the right.

But, just to be clear, that's the "phase one" of this effort.  "Phase two" is the one where I go in and click and then I want the AI to use that as the seed for where to look, which is the subject of this post.  Just wanted to clarify since the screenshot is WAY zoomed in compared to what Phase 2 would be for the main UI window.

P.S. Related side-question for which I'll open up a different thread if the answer is at all "yes," does Igor offer any sort of built-in machine learning?  I haven't seen it but that doesn't mean it's not there, and I'm not good enough with Python to try ML in that.

As you can see in my code you can get the position of the mouse click with a combination of AxisValFromPixel, s.mouseLoc.h and s.mouseLoc.v.

For browsing your craters you could make 3 buttons "Previous", "Next" and a "Accept/Reject" toggle, and bind each button to a key, for instance page up, page down and spacebar.

If you want to speed up the process, you could display the craters as a 5 x 5 grind showing 25 craters at a time. A single mouse click on each of the 1-3 bad craters would toggle them off/on and page down would load the next 25 craters. It all depends on how good your resolution has to be to accept or reject a crater.

Reducing the resolution may also make your images load faster. Depending on exactly how you write your code you can get very large differences in speed when you are working with such large images.

So, first off, thanks all for the suggestions and ideas on how to tackle this.  Wish I asked earlier!

With all the suggestions, I opted to start from the bottom first with @olelytken's code block.  I spent three hours cannibalizing and integrating much of it last night (okay, one hour was a stupid bug on my part) and it seems to be working extremely well.  I've also integrated the keyboard listener for delete to delete the last mark, arrows to scroll across the image since even these 1/600th snippets are too large to display at once (~11.5k pixels on a side), and "n" to load up the next image.  I would actually say at this point that it's doing what I want and I'm satisfied to the point I don't need to try other solutions.

That said, I want to respond to others:

@tony: Yup, the hook function works well.  It's something I've never experimented with (didn't know about, either), so it's like when I discovered multi-threading for the first time and it completely changed my world.  Regarding refining positions/points, I don't want to get quite that far unless/until there's a machine-learning component for it to learn from.  While I'm one of the only people in my field who cares about the actual rim trace/shape as opposed to just a latitude/longitude/diameter for cataloging purposes, it would take way too long to do it manually so I wold only do it to train an AI.

@johnweeks:  Right, but if you don't know all the intricacies of what can be done and the kind of interactivity Igor offers, it's hard to think about how to use that.  Going back to when I had to learn that horrid language IDL, it was a situation of you have to know what you don't know in order to search for it to learn how to do it but if you already know it then you don't need to search for it.  Regarding helping me before, I've definitely posted here before for other crater work and your particle analyzer rings a bell, but it just won't work for complicated situations like this (and that would have been for a different project since I only started this about a year ago and haven't posted to the Igor forums in that time).  There is a reason why every planetary science conference for decades has someone(s) new showing off their latest AI crater detection code and then they disappear and no one in the field uses it: Despite seeming to be simple circles, craters and the terrain they're on is complicated!  The basics behind my "low-hanging-fruit" AI is a simple circular Hough transform with some post-processing to reject most false positives (and, of course, some true positives fall into that mix).

@thomas_braun: That's sort of what I was doing with the draw tools, but it was just slow and clunky.

@aclight: Thanks.  I'll just say that based on trying the hook function, I'm going with that!  The more "true" alert box I'd like to stay away from simply because it's really fast for me to type 0/1 on the numeric keypad, or -1 to go back a crater if needed.

 

Remaining Questions [related but unrelated to this exact problem]:

  • Is Igor Pro 9 optimized for Apple's M# chips?  If not, I will probably skip this version until it is, since that will almost certainly be a faster speed increase for me.
  • Does Igor have any back-end for machine learning?  That's the next step in this but I don't know enough Python to try there and love the interactivity of Igor more.