Custom partitioning of historic data using table decorators about spark-bigquery HOT 5 CLOSED

YEs we can use table decorators using the api but this functionality is not part of the package at the moment. Sounds like what you are intending to do is a one off so I suggest using dataflow to write to the tables. no need to use spark at all if it is a one off

Table decorators would be a good feature to work on so I am marking this issue as a feature request

from spark-bigquery.

Cool thanks would be super cool - I dont know dataflow, my spark job is pretty long, and acutally reuses some of my library to do computations (like currency converting for example) - porting to a different API (Beam) would take a mear mortal like me ages - as much as I would love to get dirty with DF and be able to say I could pick it up like that and move to java (after Scala - yuk!) hehe. Wish I could.

Ill just have to make do with non partitioned, thanks for replying

p.s - still thinking about the offer, its possible if I could span out the payments longer perhaps?

from spark-bigquery.

perhaps do all your transformations and save the result to partitioned files locally? then write a simple script to upload the tables to the correct tables using the bq command line?

Yeah sure just drop me an email and we can work out the details

from spark-bigquery.

this has been fixed with version 0.2.1. @kurtmaile

from spark-bigquery.

from spark-bigquery.