Frequent inability to connect to GEO about geoquery HOT 5 CLOSED

EDIT: now compatible with getGEOSuppFiles (fixing same error)

I think the issue is with getDirListing. When processing HTML content, there can be links other than the series_matrix.txt.gz that seem to be related to determining the file size. In ronammar's case, the implicated link is '/geo/series/GSE94nnn/GSE94802/'. As a result, a download of https://ftp.ncbi.nlm.nih.gov/geo/series/GSE94nnn/GSE94802/matrix//geo/series/GSE94nnn/GSE94802/ is attempted and fails because it doesn't exist. To fix, just make sure the only links returned don't end with a forward slash (this approach also fixes the same download errors for getGEOSuppFiles). The following fixed it for me (added line ending with # !!!):

getDirListing <- function(url) {
    message(url)
    # Takes a URL and returns a character vector of filenames
    a <- RCurl::getURL(url)
    if( grepl("<HTML", a, ignore.case=T) ){ # process HTML content
        doc <- XML::htmlParse(a)
        links <- XML::xpathSApply(doc, "//a/@href")
        links <- links[!grepl('/$', links)]   # !!!
        XML::free(doc)
        b <- as.matrix(links)
        message('OK')
    } else { # standard processing of txt content
        tmpcon <- textConnection(a, "r")
        b <- read.table(tmpcon)
        close(tmpcon)
    }
    b <- as.character(b[,ncol(b)])
    return(b)
}

from geoquery.

from geoquery.

Thanks @seandavi. I'm not well versed enough in tryCatch() in R but am finding it challenging because even when a call to getGEO is successful, I still get warnings that not all columns named in 'colClasses' exist. So essentially, I can never get getGEO to run completely error or warning free. Also, the warnings are not consistent in their result: not all columns named in 'colClasses' exist results in a usable object (which has been an issue for a while https://support.bioconductor.org/p/82800/), whereas the warning status was '404 Not Found' results in nothing. Perhaps a 404 should be an error and the colClasses warning needs to be removed somehow?

from geoquery.

@alexvpickering
I also had this issue.
It seems that while accessing the matrix directory, you can get a different server (due to load balancing) and some of them have a Parent Directory link which getDirListing also tries to download.
e.g:

from geoquery.

@seandavi can you integrate @alexvpickering 's fix please?
Thank you both very much!

from geoquery.