Comments (7)
thesamet
commented on May 24, 2024
Hi @larssk , a version that supports Spark 3.3 was only released yesterday (as a fix for this ticket). Please make sure you use the right version of sparksql-scalapb according to the list here: https://scalapb.github.io/docs/sparksql/#setting-up-your-project
from scalapb.
larssk
commented on May 24, 2024
Thanks for the update ! Looks like this triggers a new issue: Task not serializable exception
Caused by: java.io.NotSerializableException: scalapb.descriptors.FieldDescriptor
Similar to issue 278 but I couldn't figure out the fix for it.
Tested with
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "1.1.0")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "1.0.6")
libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.11.13"
scalaVersion := "2.12.13"
val sparkVersion = "3.3.0"
libraryDependencies += "com.thesamet.scalapb" %% "sparksql33-scalapb0_11" % "1.0.2"
Serialization stack:
- object not serializable (class: scalapb.descriptors.FieldDescriptor, value: )
- element of array (index: 4)
- array (class [Ljava.lang.Object;, size 10)
- element of array (index: 1)
- array (class [Ljava.lang.Object;, size 3)
- field (class: java.lang.invoke.SerializedLambda, name: capturedArgs, type: class [Ljava.lang.Object;)
- object (class java.lang.invoke.SerializedLambda, SerializedLambda[capturingClass=class org.apache.spark.sql.execution.WholeStageCodegenExec, functionalInterfaceMethod=scala/Function2.apply:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;, implementation=invokeStatic org/apache/spark/sql/execution/WholeStageCodegenExec.$anonfun$doExecute$4$adapted:(Lorg/apache/spark/sql/catalyst/expressions/codegen/CodeAndComment;[Ljava/lang/Object;Lorg/apache/spark/sql/execution/metric/SQLMetric;Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;, instantiatedMethodType=(Ljava/lang/Object;Lscala/collection/Iterator;)Lscala/collection/Iterator;, numCaptured=3])
- writeReplace data (class: java.lang.invoke.SerializedLambda)
from scalapb.
thesamet
commented on May 24, 2024
Please provide a minimal example to reproduce the issue, ideally this can be done in a fork of https://github.com/thesamet/sparksql-scalapb-test or a failing test under the sparksql-scalapb/src/test/ directory in this repo.
from scalapb.
larssk
commented on May 24, 2024
Producing and consuming within this project works as expected:
thesamet/sparksql-scalapb-test@master...larssk:sparksql-scalapb-test-spark33:master
But the problem persists when reading from an existing topic. Checking..
from scalapb.
larssk
commented on May 24, 2024
Works on Spark 3.3.2. Fails on Spark 3.3.0, with java.io.NotSerializableException.
Command to execute:
spark-3.3.0-bin-hadoop3/bin/spark-submit --jars . --class myexample.RunDemo target/scala-2.12/sparksql-scalapb-test-spark33-assembly-1.0.0.jar
from scalapb.
larssk
commented on May 24, 2024
I've updated the description in order to reproduce the issue:
https://github.com/larssk/sparksql-scalapb-test-spark33/blob/master/README.md
I get the same error locally as on dataproc (which is unfortunately limited to Spark 3.3.0).
from scalapb.
thesamet
commented on May 24, 2024
Hi @larssk , thanks for creating an example - I was able to reproduce. As far as I can tell this is a Spark bug that has been fixed. We'd have to wait for dataproc to support latest version.
from scalapb.
Related Issues (20)
- Spark 3.2.1 and sparksql32-scalapb0_11 encoder error HOT 1
- [proposal] Make an option for lazy-deserialization of complex types HOT 2
- [proposal] Provide a way to inject String deduplication into the deserialization pipeline HOT 2
- Field options for types HOT 4
- Converting the result List[MAp[String,Any]] into scala.Option[com.google.protobuf.struct.Struct] HOT 3
- Feature request: add the ability to change the name of a rpc function HOT 3
- Update protobuf-java to 3.19.6 HOT 1
- Support and document buf HOT 13
- FieldMaskUtil.merge support HOT 4
- Field named `instance_of` in `oneof` causes redefinition of `isInstanceOf`
- [proposal] Allow Custom Generator Options HOT 5
- DataBricks imports `spark.implicits._` by default, is there a way to get around this? HOT 1
- trouble building under java8 on arm64 HOT 5
- Option message in option.proto fails HOT 1
- Efficiently convert bytes to strings HOT 6
- A way to set level preserve_unknown_fields for the project level. HOT 4
- Option to generate `class` instead of `case class` HOT 1
- Protobuf map with message as value won't compile HOT 3
- `collection_type` and `extends` with a `repeated` field does not compile HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scalapb.