throwing error when trying to make work locally about sparkmeasure HOT 9 CLOSED

I see you are using sparkmeasure from PySpark.
Can you please try the full example in the README.md and let me know if you still see an error?

# Python CLI, Spark 3.x
pip install sparkmeasure
bin/pyspark --packages ch.cern.sparkmeasure:spark-measure_2.12:0.17

from sparkmeasure import StageMetrics
stagemetrics = StageMetrics(spark)
stagemetrics.runandmeasure(locals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(100)").show()')

from sparkmeasure.

Hey Luca,

It worked on my work laptop - i am trying on my personal and its behaving again in sameway.

from pyspark.sql import SparkSession
import os
os.environ['SPARK_HOME'] = "C:\spark-3.2.1-bin-hadoop3.2"
spark = SparkSession.builder
.config("spark.jars.packages", "ch.cern.sparkmeasure:spark-measure_2.12:0.17").getOrCreate()

from sparkmeasure import StageMetrics
stagemetrics = StageMetrics(spark)

ERROR
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling None.ch.cern.sparkmeasure.StageMetrics.
: java.lang.NoClassDefFoundError: scala/Product$class

from sparkmeasure.

I would like to check which version of Spark you are using. I suspect you may be using Spark 2.x with scala 2.11. Can you please run: spark.version ?
Instead of setting SPARK_HOME I'd suggest the use of findspark:

import findspark
findspark.init("SPARK_HOME_HERE")

BTW I have since released sparkmeasure version 0.18.

from sparkmeasure.

Hey Luca,
Its working now.. Thanks for the help. It was issue with my java version or when i upgraded to latest sparkmeasure

from sparkmeasure.

Thanks for the feedback.

from sparkmeasure.