CSV.read slow about csv.jl HOT 7 CLOSED

Are you able to share the file? Happy to take a look at what's going on here.

from csv.jl.

Sure i sent you an invitation on your Gmail account....the file is inside that Dropbox folder.

Thanks!

from csv.jl.

@Unoqualsiasi, sorry it's taken so long to respond; my list of improvements to DataStreams & CSV was a tad long. I've been playing w/ this dataset though the last few days and it's a doozy. It's certainly non-traditional to have such a wide dataset. One question I had was if this dataset is meant to be transposed? It seems the first column is all strings like labels, and the rest of the columns are all integer codes, but the dataset has no header. Is that indeed the case? If it is, I actually have been working on a way to read in a csv file while transposing on the fly. For this csv file in particular, the read time goes down to 5-6s on my machine (vs. 70-80 on current 0.5 code).

from csv.jl.

So that dataset it's actually a small example of what you can find working in the genomic field......a lot of columns with only integers (usually 0, 1 and 2). The firs column is the ID (animal or person) and from the second column you have the SNPs. Usually you don't need labels (header) because they are stored in a different file.
Sometimes you need to transpose it, sometimes not. :D

from csv.jl.

Ok, so the CSV.TransposedSource type is now officially on master. It allows convenient reading of transposed csv data into Julia structures. It's extremely accessible, you just have to do CSV.read(file; transpose=true). For the original file here, I get timings of ~6s, which is about 3x Base.readdlm and the resulting dataset is probably much more useful (no additional transposing needed).

from csv.jl.

I am having a similar issue where CSV.read() on a 212×1005 matrix is excruciatingly slow (didn't complete in 3 minutes) while readtable() finishes in like 3 seconds. This is also a genetics dataset and I do not wish to transpose it later. I could share the dataset I'm reading as well

from csv.jl.

I was using readtable, but upgraded to the newest version of Julia. I tried to implement CSV.read to my old codes, but a relatively small data of 2K*3K took forever.

from csv.jl.