Comments (12)
DCameronMauch
commented on May 26, 2024
1
Here is a repo that demonstrates the issue:
https://github.com/DCameronMauch/TaggedType
If you change the method called by the main to the Int version, you can see that it works just fine.
Long as the base type also works.
from shapeless.
DCameronMauch
commented on May 26, 2024
Oh, if it makes any difference using the following:
Scala 2.11.12
Spark 2.4.7
Shapeless 2.3.3
from shapeless.
DCameronMauch
commented on May 26, 2024
Hello? Anyone there?
from shapeless.
joroKr21
commented on May 26, 2024
What happens if you replace the tagged type with the underlying type alias manually?
from shapeless.
alexarchambault
commented on May 26, 2024
You might have a better luck with Scala 2.12 too.
from shapeless.
joroKr21
commented on May 26, 2024
@DCameronMauch could at least give the full stack trace or a standalone reproduction we could run?
Notebooks import a lot of stuff magically and I don't know what they are.
I would bet on Encoder trying to do reflection based things and failing.
from shapeless.
DCameronMauch
commented on May 26, 2024
We will be upgrading to Spark 3.x in the upcoming months, along with Scala 2.12. So I can try that then. I'll see if I can put together an online example.
from shapeless.
DCameronMauch
commented on May 26, 2024
Here is the stack trace:
Exception in thread "main" java.lang.ClassNotFoundException: no Java class corresponding to <refinement of String with shapeless.tag.Tagged[example.DayOfWeekAsString.DayOfWeekTag]> found
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.scala$reflect$runtime$JavaMirrors$JavaMirror$$anonfun$$noClass$1(JavaMirrors.scala:1204)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1242)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToJava$1.apply(JavaMirrors.scala:1203)
at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toJava$1.apply(TwoWayCaches.scala:49)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toJava(TwoWayCaches.scala:44)
at scala.reflect.runtime.JavaMirrors$JavaMirror.classToJava(JavaMirrors.scala:1203)
at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:194)
at scala.reflect.runtime.JavaMirrors$JavaMirror.runtimeClass(JavaMirrors.scala:54)
at org.apache.spark.sql.catalyst.ScalaReflection$.getClassFromType(ScalaReflection.scala:726)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$dataTypeFor$1.apply(ScalaReflection.scala:107)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$dataTypeFor$1.apply(ScalaReflection.scala:88)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:929)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:49)
at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$dataTypeFor(ScalaReflection.scala:87)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor$1$$anonfun$8.apply(ScalaReflection.scala:658)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor$1$$anonfun$8.apply(ScalaReflection.scala:651)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor$1.apply(ScalaReflection.scala:651)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor$1.apply(ScalaReflection.scala:471)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:929)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:49)
at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$serializerFor(ScalaReflection.scala:471)
at org.apache.spark.sql.catalyst.ScalaReflection$.serializerFor(ScalaReflection.scala:460)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:71)
at org.apache.spark.sql.Encoders$.product(Encoders.scala:275)
at org.apache.spark.sql.LowPrioritySQLImplicits$class.newProductEncoder(SQLImplicits.scala:248)
at org.apache.spark.sql.SQLImplicits.newProductEncoder(SQLImplicits.scala:34)
at example.Application$.tryDayOfWeekAsString(Application.scala:33)
at example.Application$.main(Application.scala:6)
at example.Application.main(Application.scala)
from shapeless.
DCameronMauch
commented on May 26, 2024
When I run it against the Int version of DayOfWeek, I get this expected output:
+---+---------+
|id |dayOfWeek|
+---+---------+
|1 |1 |
|2 |3 |
|3 |5 |
+---+---------+
from shapeless.
DCameronMauch
commented on May 26, 2024
New piece of information: I created a "spark3" branch on that repo. Using Spark 3.1 and Scala 2.12. In this environment, the string based tagged type works as expected. So it's either a Spark 2.4 and/or Scala 2.11 thing. I feel like it's still worth investigating, because there are a lot of people stuck on Spark 2.2 or 2.4 with Scala 2.11. We hope to upgrade to the latest in the next few months, so at least the issue will be resolved for us.
from shapeless.
DCameronMauch
commented on May 26, 2024
Tried a few more combinations. The issue appears to be with Spark 2.4. Even with Scala 2.12, it still fails. But soon as I upgrade to Spark 3.0, it starts working. Couldn't test Spark 3.0 with Scala 2.11, as that's not supported.
from shapeless.
joroKr21
commented on May 26, 2024
The issue is with Spark SQL - it doesn't work with refined types (which is what @@ translates to). Here is the offending code:
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala#L88-L112
It works for primitives because they are special cased in dataTypeFor with isSubtype - so subtypes of primitives are basically treated as primitives. Unfortunately Spark is not very good at offering extension points and I don't think you can define a custom DataType for @@. But if you consider using https://github.com/typelevel/frameless it does let you define custom encoders: http://typelevel.org/frameless/Injection.html
from shapeless.
Related Issues (20)
- implicit summoning of records.Keys for singleton subtype of HList fail sporadically HOT 3
- Witness path-dependent type `T` lose refinement HOT 3
- Witness singleton type automatically erased by compile-time type inference HOT 3
- migrate to GitHub Actions HOT 1
- Implicit is not returned from the cache but rather from current scope HOT 3
- Migrate to GitHub actions HOT 1
- `Default.AsRecord` causes `StackOverflow` in a path-dependent type's companion object HOT 5
- `ops.coproduct.Reify` broken HOT 1
- performance issue with combined `Length` and `ToSizedHList` implicit derivation HOT 3
- Generic in shapeless 2.3.5+ not working for classes with context bounds if some implicit value is present HOT 2
- Generic.Aux compiles but errors with ClassCastException at runtime HOT 3
- Shapeless 2.3.5+ can't provide implicit for Generic.Aux HOT 5
- _0 as defined is somehow causing extreme compile times HOT 1
- Possible derivation regression since shapeless 2.3.8 HOT 3
- Strange behavior when using -release 8 scalac option HOT 3
- Fix annotations with type parameter
- Shouldn't `KeyTag` be an abstract type rather than a trait? HOT 4
- `Generic` is not materialized in macro-generated companion object of nested case class HOT 1
- MkFieldLens.mkFieldLens returns a derived type instead of type parameter
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from shapeless.