Comments (3)
yogesh-0586
commented on May 28, 2024
On spark-shell running DML query run fast, but on spark-submit running in yarn and cluster mode it took almost 2-3 minutes for single query
from spark-bigquery.
samelamin
commented on May 28, 2024
This isn't a bug but more of a performance issue, behaviour is very different between a spark shell and spark submit depending on how you are using spark submit
Best to add some form of telemetry to this ticket to validate that it's the connector not yarn or spark adding this overhead
from spark-bigquery.
yogesh-0586
commented on May 28, 2024
@samelamin Thanks for your reply, I found that where was issues, it's from spark side
from spark-bigquery.
Related Issues (20)
- class cast exception has occurs (Double cannot be cast to Float) HOT 3
- Using spark-bigquery connector in AWS EMR Zeppelin HOT 4
- java.lang.ClassCastException: java.lang.Long cannot be cast to java.sql.Timestamp HOT 1
- Write to bigquery using DataframeWriter HOT 2
- runDMLQuery on AWS Glue throws an exception because of Jersey libraries conflict HOT 1
- read table second time show error Conflict occurred creating export directory HOT 1
- Error when changing zone to something other than EU/US HOT 3
- Utilize Bigquery Storage API HOT 2
- getting java.lang.NoSuchMethodError on saveAsBigQueryTable HOT 1
- Exception in thread "main" java.lang.NoSuchMethodError: HOT 14
- Struck with error py4j.protocol.Py4JJavaError: An error occurred while calling o39.saveAsBigQueryTable. : java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;) HOT 6
- Give content of JSON credentials instead of path to it HOT 2
- Big query export from GCP to AWS S3 using Spark HOT 2
- saveAsBigQueryTable exception with StructType column HOT 1
- Export FS must derive from GoogleHadoopFileSystemBase HOT 1
- writing to bigquery from dataproc HOT 1
- Method saveAsBigQueryTable([class java.lang.String]) does not exist: Issue persists even after shading guava dependecies HOT 4
- Conflict occurred creating export directory already exists HOT 1
- 404 not found: Job <job name> not found error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-bigquery.