NIDAQ Waveform Generator unstable with very large wave

I am having an issue with the waveform generator in NIDAQ tools MX.  Previously, I had been able to set up the Scan control panel to capture one data channel at 50 kHz x 24average = 1.2 MHz, triggered off the waveform generator (or rather ao/starttrigger from the NI box) running a single wave with a period of 43sec at a rate of 1 MHz (43 million points in wave) for 9 cycles.

Now, days later, when I try to run this same configuration, the waveform generator either outputs for part of one period then stops without errors, while the NI box continues to output the DC value of the waveform whereever the the waveform generator stops, or the waveform generator procedure hangs and I have to hit the abort button a bunch of times and wait awhile or force quit.

I suspect this is some sort of excessive resource use issue since I can still output a less demanding but similar waveform at 100kHz, but this worked fine before a bunch of times, this is well within the capabilities of the NI box, and I don't see a spike in the CPU (maybe 1 to 5%useage) or memory (maybe 40 to 60%) useage in the windows task manager.  Is there something I can do to troubleshoot this?

I'm also careful not to have a live updating graph window or anything else going that would contribute to using resources.

It's hard to say what's going on here, and I can't totally rule out a bug in NIDAQ Tools MX. Have your re-started Igor anywhere in the story you outlined? Or re-started your computer?

Clearly, your waveform is larger than anything that will fit in an onboard buffer on your DAQ device, so there will be lots of memory transfers going on. My guess is that the waveform is stopping on a buffer underrun- the system simply isn't getting the data out of system memory and transferring to the DAQ device quickly enough. Why it would have worked previously and stopped later is a mystery. If you have not restarted Igor in the meantime, it's possible that memory fragmentation might contribute to the problem. As you make Igor waves and kill them and make new ones, over time Igor's memory space can be broken into small bits making memory access slower. The only way to clean that up is to re-start Igor.

What version of NIDAQ Tools MX are you using? If you are using the waveform generator panel provided with the package, it should report the error that causes it to stop. In the most recent versions, it creates an Igor notebook window containing error information. That might shed light on what's happening.

Thank you for writing back.  I suspected the answer might not be straightforward since I am pushing the computer kind of hard and the devices have to talk to each other very quickly.  I noticed that even when it was working correctly, after I clicked "start" on the waveform generator, there would be about 0.5-1 second before the waveform would actually output from the box and trigger the scan control.  I did try restarting Igor, and then the entire computer, pretty early on when the problem appeared, to no avail, so perhaps the memory fragmentation issue you mention is not solely to blame.

I think the version I am using is recent. It's the demo version from the website: NIDAQ Tools MX - DemoSetup.exe with product version 1.1.2.0 from the properties panel. I am not sure whether I am using a 64 or 32 bit version. I have enough useful data (this test type is important, but other important tests with somewhat slower output rates still work flawlessly with the NIDAQ tools) that I should be able to convince my supervisor to purchase it for me prior to the trial running out in a couple weeks.  But I believe this will just stop the warning message about the trial version from appearing at startup, not lead to downloading a new version or anything.

You mention the error reporting in a notebook, which I have seen when I do something silly, like choose an output rate above the maximum for the device, but there is no error involved here.  One of two things happens:

1) Igor is busy for several minutes, but nothing is ever output from the NI box and I eventually have to force quit or hit the abort button a few times and wait 30 seconds or so.  If the abort works, it just gives a message saying there was a user abort.

2) The waveform generator outputs a partial cycle, then the start button is active again and the "ready" indicator turns green. There are no error messages and, as far as I can tell, Igor thinks everything worked fine.  The NI box continues to output a DC value corresponding to whenever the function generator stopped.  If the function generator manages to output the first cycle correctly, it seems to make it through the subsequent cycles with no problem, although I only have one example of this since the problem started appearing.

(2) is much more common than (1) as an outcome.  Does any of this provide any clues about troubleshooting I could pursue?

It's curious that you don't see any error message in the 2) case. The Waveform Generator panel uses the DAQmx_WaveformGen error hook (flag /ERRH flag). The fact that the Stop button goes to Start means that either the /ERRH error hook is running, or the /EOSH end hook is running. If it is the /EOSH hook, that means that NIDAQ Tools MX didn't detect a problem, or that my code didn't detect it. I would expect that a buffer underrun would result in an error, but it is very hard to test on purpose.

It's possible that the use of a USB device contributes to the problem. USB is designed to be fast, but not truly real-time. National Instruments has apparently gone to great lengths to make it work reliably, and mostly it does. But I do wonder if perhaps pushing that much data back and forth over USB is not always reliable. Do you have other devices using the same USB port?

Does your output waveform really need to be that long? What sort of waveform is it? If you have many cycles of a periodic waveform in your wave, if one period would fit in 8192 samples that would allow all the waveform data to reside in the device so that the USB isn't required to transfer output data all the time. That would probably totally eliminate any buffer underrun situation.

I'm not really able to try to simulate your situation here, since I have a USB-6341. It's limited to 500K samples/sec input, and 900k samples/sec output. The lower rates wouldn't stress the USB as much.

 

Thank you for the context. This was the only device on the USB port.  I know this particular box uses the USB 2.0 standard, even though the port itself is 3.0.  Unfortunately, the waveform does need to be that long, due to what it is trying to do, and I have tried to find a way to shorten it, to no avail so far.  Even if I could shorten it, I would still need the rate to be very high, at least for one section of the wave, since the signal is moving a physical piezo, in this case very fast with varying speed, and the measurement requires the movement to be smooth, since I am measuring a velocity dependence.  If I measure the velocity-dependence up to a slower max speed, I can slow down the rate to 100 kHz, and this works great. In case you are interested, I have attached an image of the waveform for the very fast case.

For the time being, I'll probably resample the fast waveform to start slowing down the rate some to see when the problem goes away.  It's a judgement call, but I suspect I can get away with dropping the rate by half or so without messing up the measurement.  It'd be nice to push the box to the max since going faster gives more experimental insight, but I've already pushed it to a very useful speed range.

Should I try putting a breakpoint in /EOSH or /ERRH to confirm that whatever is happening isn't causing an error?  Or is that already confirmed just by the fact that Igor isn't showing me any errors now?

I guess you want the sample rate really high to reduce the "stepiness" of the output. Naturally, the digitization error will be greatest during the high-rate section at the start and end of each half-period. Unfortunately, you can't optimize it by having slower sampling where the change rate is lower.

Are you doing the waveform and scanning in the background? It might be that you would be better off using a synchronous call for the waveform. To do that, you would need to set up the scan as a background scan with the trigger on the waveform. Then set up the waveform not as background. When it starts, it will trigger the scan. The downside is that you can't see what's happening until it's all over.

Another possibility is that Igor's event loop isn't running fast enough. In Igor 8 you can change the event loop period with the SetIdlePeriod operation. Be sure the read the warnings in the documentation.