Struck with error py4j.protocol.Py4JJavaError: An error occurred while calling o39.saveAsBigQueryTable. : java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;) about spark-bigquery HOT 6 CLOSED

Unable to import Module..
import com.samelamin.spark.bigquery._
Import ERROR
in python files..

from spark-bigquery.

This sounds like a bug that was fixed with the latest release, can you confirm what version you are using?

from spark-bigquery.

I picked 2.6.0 and built an uber jar. Still doesn't work. Read and Write, both throwing same error
My Jar size: 15710607 Apr 29 15:49 sparkbigquery-0.0.1-SNAPSHOT.jar (this is a wrapper jar. I built it so that I can have a big uber jar & incl all)

Read:
scala> val df = spark.sqlContext.read.format("com.samelamin.spark.bigquery").option("tableReferenceSource","projectid:schema.table").load()
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68)
at com.samelamin.spark.bigquery.BigQueryRelation.getConvertedSchema(BigQueryRelation.scala:19)
at com.samelamin.spark.bigquery.BigQueryRelation.schema(BigQueryRelation.scala:13)
at org.apache.spark.sql.execution.datasources.LogicalRelation.(LogicalRelation.scala:40)
at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:389)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:125)

Write:
scala> avrodf.saveAsBigQueryTable("projectid:schema.table")
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68)
at com.samelamin.spark.bigquery.BigQueryDataFrame.saveAsBigQueryTable(BigQueryDataFrame.scala:40)
... 50 elided

guava: 26.0

jdk1.8

Please help us. This seems to be a great api. very promising to use and easiness.

from spark-bigquery.

This sounds like a bug that was fixed with the latest release, can you confirm what version you are using?

2.6.0 also has this error. ShowStopper.

from spark-bigquery.

I was able to get this resolved by shading the google libraries.

 <configuration>
   <relocations>
     <relocation>
       <pattern>com.google</pattern>               
         <shadedPattern>shaded.guava</shadedPattern>
           <includes>
             <include>com.google.**</include>
           </includes>
           <excludes>
             <exclude>com.google.common.base.Optional</exclude>
             <exclude>com.google.common.base.Absent</exclude>
             <exclude>com.google.common.base.Present</exclude>
             <exclude>com.google.cloud.**</exclude>
           </excludes>
     </relocation>
   </relocations>
 </configuration>

from spark-bigquery.

Cheers for adding the example @ameyamahajan I would really appreciate it if you add a ToDo section in the readme :)

from spark-bigquery.