Reading CSV file easily and quickly

[milfredo]

I access csv data files. Each record in a file has 1435 data slots. Right now I read each data point into an array and then pull out the data from that array that I need for my program. This seems to be the long way to do it, cause the file could have anywhere from 80 to 150 records each 1435 data slots. How can I cut down and make this quicker? Most of the 1435 data points I don't need or use. So I am burning a lot of time going through each record one data point at a time. That becomes a problem because I have anywhere from 10 to 32 data files to read at a time.


 Could use some help.


Thanks,

 Milfredo

[Chris Iverson]

If you know the numbers of the data slots you need, you can use the word$() command to pull those out quickly.

Like, if you just need the 6th, 25th, and 80th:

open "mytest.csv" for input as #file
while eof(#file) = 0
   line input #file, csvLine$
   dat1$ = word$(csvLine$, 6, ",")
   dat2$ = word$(csvLine$, 25, ",")
   dat3$ = word$(csvLine$, 80, ",")
   print dat1$, dat2$, dat3$
wend
close #file

Note that this does NOT parse quotes, so if you need to process a file that has commas in the data as well as a separator, you'll need to write more code to handle those cases.

[milfredo]

Thank you so much. I will give it a try.   I just noticed this goes till the end of the file. But how do I detect the end of the first record? I will need to get data item 80 for every record. Is that taken care of by the line input? 


Milfredo

[Chris Iverson]

If it's a standard CSV file, it should be.

CSV files (should) have each record on a separate line, with commas separating individual values.

[Rod]

A lot of files are saved as CSV without any real need because the data is actually fixed width. If this were the case you could simply use line input then parse out the data with mid$() If there are so many data elements I would be surprised if they were variable length. One mid$() parse would certainly speed your program.

[milfredo]

I have the field map for my data files and so no problem with getting the data out. I was looking for a much quicker way of accessing  the data and you guys have cleared things up. Thanks so much.


Once I get some coding done I will let you know if I got it. Been sick so no coding right now.


Milfredo