Interfacing with C functions from XOPs with unsigned long long return values
thomas_braun
Fri, 05/04/2018 - 03:13 am
I'm thinking about adding a XOP which uses https://github.com/Cyan4973/xxHash to calculate hash sums. In my tests it is much faster than WaveCRC.
One problem is that the hash function [1] returns an unsigned long long which is a 64bit integer on Windows 64bit. And I can't return that from an XOP function AFAIK.
The following work arounds spring to mind:
- Return a string with the stringified hash
- Use a 64bit integer wave for transfer
- Return a struct with pass-by-reference with a 64bit integer
But all include additional work and make for an ugly interface.
Any other options I have missed?
I think if I do
int64 val = FancyHash(myWave)
where FancyHash returns a 64bit interger as a double I don't get what I expect as Igor will convert from double to integer.
[1]: https://github.com/Cyan4973/xxHash/blob/3064d42e7d74b0921bdd1818395d9cb…
It sounds like this would be worthwhile only if calculating the hash takes a significant portion of the overall processing time.
You have that about right.
An additional option is to return a complex double with half of the 64-bit integer in the real part and half in the imaginary part.
Another is to return half of the 64-bit integer via one pass-by-reference double and the other half via another pass-by-reference double. The function result would be an error code.
Another is to return a wave reference to a free 64-bit integer wave, but making a wave is not particularly fast.
May 4, 2018 at 05:19 am - Permalink
In $bigApplication we quite often calculate large waves from various parameters including smaller (> 100k points) waves. In order to bypass this longish calculation I've created a cache for these waves. The key to store and retrieve the waves into/from the cache is the crc of all input parameters and also the WaveCRC of these input waves. And if Igor's function profiling is not lying we are currently spending 15% of our time in calculating the WaveCRC. My preliminary tests with the new hash function could get that down already.
I always forget about complex doubles. Thanks for the hint. This sounds like the way to go.
May 4, 2018 at 05:47 am - Permalink
Since we're exposing WaveModCount in the XOP toolkit, we could probably add a built-in WaveModCount function that does the same thing. Then maybe you wouldn't need an XOP.
May 4, 2018 at 09:22 am - Permalink
Abstract from our private conversation: This does not work as I need a value which is unique for the wave contents.
Thanks to hint of a coworker I've found that the SSE4.2 instruction set supports CRC32 [1, 2], altough not with the same polynominal as
WaveCRC.The attached graph compares the speed of WaveCRC, xxhash64 and crc32_hw for various wave sizes.
And as expected no algorithm can beat silicon.
I've used https://bitbucket.org/robertvazan/crc32c-hw/src and https://github.com/Cyan4973/xxHash and IP8 beta from today.
[1]: https://en.wikipedia.org/wiki/SSE4
[2]: https://software.intel.com/sites/default/files/m/8/b/8/D9156103.pdf
May 18, 2018 at 04:20 am - Permalink