Comments (4)
Hi
Looks like you need to create an uber jar because the connector on its own needs the google client. Most clusters already have that but sounds like you need to load it onto zeppelin specifically
I would suggest creating a fat jar and seeing if you can get spark to run a sample application on the EMR, once that works it should be simpler to port it to zeppelin
from spark-bigquery.
Hi. I created an uber jar with google client library using sbt.
libraryDependencies ++= {
val sparkVer = "2.2.1"
val sparkbqVer = "0.2.4"
Seq(
"org.apache.spark" %% "spark-core" % sparkVer % "compile" withSources(),
"org.apache.spark" %% "spark-sql" % sparkVer % "provided", //% "compile" withSources(),
"org.apache.spark" %% "spark-hive" % sparkVer, //% "provided" withSources(),
"com.github.samelamin" %% "spark-bigquery" % sparkbqVer,
"com.google.api-client" % "google-api-client" % "1.23.0"
)
}
This is the error I am getting when i submit my spark job with spark submit command ,
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.cloud.hadoop.io.bigquery.BigQueryStrings.parseTableReference(BigQueryStrings.java:68)
at com.samelamin.spark.bigquery.BigQueryRelation.getConvertedSchema(BigQueryRelation.scala:19)
at com.samelamin.spark.bigquery.BigQueryRelation.schema(BigQueryRelation.scala:13)
at org.apache.spark.sql.execution.datasources.LogicalRelation$.apply(LogicalRelation.scala:77)
at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:424)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:146)
at dataload.pull_gbq_data$.main(pull_gbq_data.scala:18)
at dataload.pull_gbq_data.main(pull_gbq_data.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
from spark-bigquery.
That is a different error @Jeeva-Ganesan and it has to do with Guava
You need to shade it into the uber jar, if you google it you will see what I mean
I think anything about guava 18 should fix it
from spark-bigquery.
Yes. I tried that, I changed my build file like this,
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.guava.**" -> "my_conf.@1")
.inLibrary("com.google.guava" % "config" % "23.6")
.inProject
)
libraryDependencies ++= {
val sparkVer = "2.2.1"
val sparkbqVer = "0.2.4"
Seq(
"org.apache.spark" %% "spark-core" % sparkVer % "provided", //compile" withSources(),
"org.apache.spark" %% "spark-sql" % sparkVer % "provided", //% "compile" withSources(),
"org.apache.spark" %% "spark-hive" % sparkVer, //% "provided" withSources(),
"com.github.samelamin" %% "spark-bigquery" % sparkbqVer % "compile",
"com.google.api-client" % "google-api-client" % "1.23.0" % "compile",
"com.google.guava" % "guava" % "23.6"
)
}
Still got the error, ended up downloading latest guava jar and placing it in spark jars folder (deleting the existing one). Then it worked.
from spark-bigquery.
Related Issues (20)
- class cast exception has occurs (Double cannot be cast to Float) HOT 3
- java.lang.ClassCastException: java.lang.Long cannot be cast to java.sql.Timestamp HOT 1
- Write to bigquery using DataframeWriter HOT 2
- runDMLQuery on AWS Glue throws an exception because of Jersey libraries conflict HOT 1
- read table second time show error Conflict occurred creating export directory HOT 1
- Error when changing zone to something other than EU/US HOT 3
- DML query drop and create table takes time HOT 3
- Utilize Bigquery Storage API HOT 2
- getting java.lang.NoSuchMethodError on saveAsBigQueryTable HOT 1
- Exception in thread "main" java.lang.NoSuchMethodError: HOT 14
- Struck with error py4j.protocol.Py4JJavaError: An error occurred while calling o39.saveAsBigQueryTable. : java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;) HOT 6
- Give content of JSON credentials instead of path to it HOT 2
- Big query export from GCP to AWS S3 using Spark HOT 2
- saveAsBigQueryTable exception with StructType column HOT 1
- Export FS must derive from GoogleHadoopFileSystemBase HOT 1
- writing to bigquery from dataproc HOT 1
- Method saveAsBigQueryTable([class java.lang.String]) does not exist: Issue persists even after shading guava dependecies HOT 4
- Conflict occurred creating export directory already exists HOT 1
- 404 not found: Job <job name> not found error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-bigquery.