Timeouts when running with paws.database 0.1.10 about noctua HOT 7 CLOSED

noctua utilises the services of aws athena, s3 and glue. This corresponds to paws.analytics and paws.storage. To find out what is the most likely culprit of this error will break down the dbGetQuery call.

dbGetQuery breakdown:

dbGetQuery calls the following noctua methods:

• dbSendQuery
• dbStatistics
• dbFetch
• dbClearResult

dbSendQuery calls:

• paws.analytics::athena -> start_query_execution. This is to start an AWS Athena query.

dbStatistics calls:

• paws.analytics::athena -> get_query_execution. Get memory usage from Athena.

dbFetch calls:

• paws.analytics::athena -> get_query_execution. Get Athena execution Status
• paws.analytics::athena -> get_query_results. Get Athena column class so that it can be passed back to file parsers
• paws.storage::s3 -> get_object. Get Athena result

dbClearResult calls:

• paws.analytics::athena -> get_query_execution. Get Athena execution Status
• paws.storage::s3 -> delete_object. Remove Athena S3 result file from S3. Note only called when cache equals 0

from noctua.

Thanks for diagnosing the bug, which we inappropriately introduced as a default timeout in the last release of paws.common. Sorry about that. The latest version (0.3.8) with a fix (no timeout) is now on CRAN.

from noctua.

Hi @davidski thanks for letting me know. Does this error happen with the DBI interface? Or is it just the dplyr interface?

I will have a little look at paws.database to see what the change could be and how to fix it :)

from noctua.

Thanks for the quick response! Yes, this problem occurs when making a DBI-style query as well. ☹️

from noctua.

@davidski how long did this query run before it timed out?

from noctua.

As the statistics of the query has been returned (Info: (Data scanned: 14.09 MB)) the best culprit would be dbFetch. I believe it is paws.storage::s3 -> get_object causing this issue. And a possible change to paws.common. I will update issue paws-r/paws#371 accordingly.

from noctua.

@davidkretch thanks again. I will close this ticket.

from noctua.