Intern: in over my head; *.dat files and waves
RochaJM79
Fri, 08/01/2008 - 01:32 pm
Basically, the situation:
We monitor wells for contaminants and in doing so we keep track of many factors using Drucks. examples of some of the data are date d/l'ed, serial date, slope, offset, druck position elevation, druck raw data, water level raw data, etc. etc.
The information is downloaded and immediately stored in *.dat files saved in a delimited format separated by comma's.
Here is where I am truly confused, we save the data according to the monitoring wells name, ie 3984.dat, but that is just for one month. For every different month there is a different folder and each folder basically has the same names, with that months information on the monitoring wells.
I need to set IGOR up in such a way that as each well has a wave and graph, updated monthly, with as little manual copying and pasting as possible.
My supervisor basically wants it so that we have all files in say C:/drucks/wells/2008/ and each month has its files like C:/drucks/wells/2008/june/*.dat.
How can I get IGOR to constantly update the wave per well and keep track of month and year?
Please Help!
I think you need to ask more specific questions. Something more bite-sized.
August 1, 2008 at 02:42 pm - Permalink
I'm guessing your wells are numbered, and 3984 is one such number. So I guess you have well 3984's June 2008 data in C:/drucks/wells/2008/june/ as 3984.dat along with, say 1234.dat and 5678.dat, etc.
I'm further guessing that you want a graph of data for a particular well's factor displayed on the Y axis versus the date on the X axis, and you want to update periodically, continuing where you left off.
It seems to me that your Igor experiment can simply record the date that was last examined (see date2secs), and index through the folders (IndexedDir) looking for folders that are newer, load each file in the folder (IndexedFile), extract the factors, locate the wave associated with that well and factor, and append the factor (and possibly append the date to a shared X wave) using InsertPoints and an assignment to the appended point (array element).
August 1, 2008 at 03:01 pm - Permalink
Okay, so I will be looking up what you have given me to work with. But I've been trying to figure out, the date that is in the *.dat files is separated into 3 columns. year, julian date, and time. The time is taken at 0000 and 1200 hrs daily. How would I get it to recognize the julian date.
I think thats all I can deal with right now.... appreciate the help, I will be working on that, and if there are any suggestions or snippets to make this easier, I would really appreciate!
Thanks
August 1, 2008 at 03:39 pm - Permalink
You need to convert Julian date to Igor date/time format which is seconds since January 1, 1904.
If you execute Print/D dateToJulian(1904, 1, 1), Igor prints 2,416,481. I believe this is the number of days from the start of the Julian date system to January 1, 1904. Therefore, a Julian date can be converted to an Igor date/time value like this:
Variable julianDate
Variable secondsInADay = 24*60*60
Variable igorDateTime = (JulianDate - 2416481) * secondsInADay
return igorDateTime
End
You can test this like so:
You can further test by converting today's date to Julian and then to Igor date/time and finally to a date string, like this:
2454683
Print/D JulianDateToIgorDateTime(2454683) // Convert to Igor date/time
3300652800
Print Secs2Date(3300652800,1) // Convert Igor date/time value to date string
Monday, August 4, 2008
So far, everything looks good. However, I have read that Julian dates start at noon whereas Igor dates start at midnight so you might need to add a 12-hour offset to JulianDateToIgorDateTime. Since it is getting late, I will leave that as an exercise for the reader.
Assuming you have a valid JulianDateToIgorDateTime function, you can convert a wave containing Julian dates to a wave containing Igor date/time values like this:
You can then add your time wave, which I assume contains seconds since midnight, like this:
August 4, 2008 at 07:51 pm - Permalink
August 5, 2008 at 03:31 pm - Permalink
The delimited text file loader here and the SpXZeigR display package here might be a start. Based on what I understand of your needs, here is what the combination can do right now ...
... and here is what it cannot do right now ...
All of the above functions could be obtained by modifying the delimited text loader with specific loading and processing options to suit your needs.
Let me know if you might want to use this as a base. I can then recommend how you might start by modifying the existing delimited text file loader.
--
J. J. Weimer
Chemistry / Chemical & Materials Engineering, UAH
August 6, 2008 at 05:27 am - Permalink
I have spoken with my boss to work this in such a way that it is not 100% automatic.
New Goal:
17 Wells being monitored
readings recorded in CSV format
2 readings per day at 0000 and 1200
each day recorded in julian format
each month data is collected and saved
originally I had planned to save all the montly data in different folders ie: 2007/mar/3747.dat ; 2007/apr/3747.dat, however, I realized how difficult that was going to be.
New Idea. I am going to combine all .csv files into one large csv file. so instead of all the folders of years and months it will now be one folder of wells 3747/3747.csv and each additional month the lab tech will just copy and paste new information to the bottom of the file.
Next issue I'm working out, there are several waves in each .csv. I know 3 are automatically year, julian date, and time (0000, or 1200). I need the graph to display month/day/time like - 03/04/2003-1200 or something similar and easy to view. I'm figuring I'm just going to figure a formula that converts the julian date and adds it to the time to create a new wave and then have that wave graphed, is that correct?
The remaining waves are D elevation, D reading, and W elevation along with slope and offset. I used the slope and offset previously in excel to create a corrected W reading and D reading. Once again I'm going to assume that is going to create 2 new waves, D Reading and W Reading, those will be plotted.
Now at this point each well should have one graph and one chart, that is from several years back to now. Next issue I'm working on is I have an excel file that has all the manual recordings of the well (to compare to the automated data) that is taken only once per month. That is setup with a column as date and another column as depth reading. That will be plotted also, as just a point on the graph.
So thats it, each well is going to have a graph with Corrected D elevation, Corrected W Elevation, and W reading from a large *.csv file, I will have from an excel file manually recorded data added to the chart as a wave and plotted on the graph as a point.
Will I be able to leave it so that someone who works in this office simply opens Igor, clicks on a filename, and each graph and corresponding chart for all 17 wells opens up with no work required on their part? Or will they have to select a filename that opens up a chart and graph for just one well?
I wish I had had a lot more time to work on this and learn this, but in between all the different assignments I am completing here I cannot dedicate enough time to learn and program this as I should be able to.
Sorry I was so vague in the past as I was only getting a trickle of information at a time. Thanks.
August 8, 2008 at 03:23 pm - Permalink