Simple setup for it-hadoop-client about cmsspark HOT 11 OPEN

I don't really see any actions here I need to do. You can do this or any other stuff if you want. What exactly do you require from CMSSpark?

In another ticket #24 I tested your code where I didn't specified custom configuration and it works on it-hadoop-client. Therefore please provide concrete action/issue you're asking to fix or close this ticket.

from cmsspark.

Its more of a note for people here that may find useful. I'm making suggestions to simplify the codebase.

from cmsspark.

@nsmith I tried it on SWAN python notebook with analytix/spark/hadoop, but it can not find a class:
Py4JJavaError: An error occurred while calling o82.load.
: java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.avro.AvroFileFormat. Please find packages at http://spark.apache.org/third-party-projects.html
I think I passed CMSSpark because it seems to have databriks.spark.avro, too.
I tried it on the command line, but have the same error:
bash-4.2$ spark-submit --packages com.databricks:spark-avro_2.11:4.0.0 test.py 2>&1 | grep Fail
: java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.avro.AvroFileFormat. Please find packages at http://spark.apache.org/third-party-projects.html

My spark env looks like:
SPARK_HOME=/cvmfs/sft.cern.ch/lcg/releases/spark/2.4.0-cern1-6f44e/x86_64-centos7-gcc7-opt
SPARKMONITOR_KERNEL_PORT=36803
SPARK_CLUSTER_NAME=analytix
SPARK_CONF_DIR=/cvmfs/sft.cern.ch/lcg/etc/hadoop-confext/etc/spark.analytix/conf
SPARK_PORTS=32832,40308,36371,34794,36297,39138
SPARK_USER=bockjoo
HADOOP_TOKEN_FILE_LOCATION=/spark/hadoop.toks
PYSPARK_PYTHON=/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-c333c/x86_64-centos7-gcc7-opt/bin/python
PYSPARK_DRIVER_PYTHON=/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/bin/python
SPARK_CONFIG_SCRIPT=/cvmfs/sft.cern.ch/lcg/etc/hadoop-confext/hadoop-setconf.sh
SPARK_LOCAL_IP=172.17.0.5
SPARK_DIST_CLASSPATH=/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/etc/hadoop:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/common/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/common/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/hdfs:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/hdfs/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/hdfs/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/yarn/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/yarn/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/mapreduce/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/mapreduce/
In [2]:

Do you know how to pass the needed packages in SWAN?
Should I add the classpath for org.apache.spark.sql.avro.AvroFileFormat?
If so, which where is the jar file for that?

from cmsspark.

from cmsspark.

Have you tried not using CMSSpark at all? If you just follow https://hadoop-user-guide.web.cern.ch/hadoop-user-guide/getstart/client_cvmfs.html it works fine for me on lxplus. It should work on SWAN if you include the 'CMSSpark options' checkbox. (you don't need to setup the session in SWAN though)

from cmsspark.

Actually I lied, it works fine on the edge nodes but not lxplus.

from cmsspark.

@vkuznet Thanks Velentine! I will try to set up avro jars following run_spark script.
@nsmith Yes, I tried checking CMSSpark options, but it did not work, i.e., when I checked the avro jars from SWAN, I got this:
!for p in $(echo $SPARK_DIST_CLASSPATH | sed 's#:# #g') ; do echo $p ; done | grep avro
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc7-opt/share/hadoop/common/lib/avro-1.7.4.jar
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc7-opt/share/hadoop/mapreduce/lib/avro-1.7.4.jar

I will see if setting the jar like run_spark will help.
By the way, I wanted to read /project/awg/cms/job-monitoring/avro-snappy/year=2019/month=5/day=1/*.avro to correlate events with /project/monitoring/archive/xrootd/raw/gled/
Just so you know.

from cmsspark.

@bockjoo if you're interested in a high-level comparison of these popularity sources, I can forward you some material.

from cmsspark.

@nsmith Yes, please. Thanks!

from cmsspark.

This works on SWAN Bleeding Edge ( @nsmith I am not sure if you meant this Edge or k8 edge or something else ):
from pyspark import SparkContext, SQLContext, SparkConf
from pyspark.sql import SparkSession
conf = SparkConf().setMaster("local").set("spark.driver.memory", "1g").set("spark.executor.memory", "1g").set("spark.jars.packages","org.apache.spark:spark-avro_2.11:2.4.0")
sc = SparkContext(conf = conf)
spark = SparkSession(sc)
readavro = spark.read.format("avro")
df_job = readavro.load("/project/awg/cms/job-monitoring/avro-snappy/year=2019/month=5/day=1/*.avro")
df_job.printSchema()

from cmsspark.

@nsmith- , @bockjoo , this is what I suspected, even though the usage is trivial as Nick pointed out in his first post, the number of use-cases when you'll look at different data-sources will grow and eventually to accommodate all of them you'll converge on something CMSSpark is aiming to.

I'm not against this simple solution, but you should keep in mind that it will be adopted/modified to every source we have on HDFS. Unfortunately, due to heterogeneous sources/formats/structures there is no simple "solution" and I rather in favor to keep such details hidden from end-users and simplify access to all sources via common framework.

from cmsspark.