Comments (11)
I don't really see any actions here I need to do. You can do this or any other stuff if you want. What exactly do you require from CMSSpark?
In another ticket #24 I tested your code where I didn't specified custom configuration and it works on it-hadoop-client. Therefore please provide concrete action/issue you're asking to fix or close this ticket.
from cmsspark.
Its more of a note for people here that may find useful. I'm making suggestions to simplify the codebase.
from cmsspark.
@nsmith I tried it on SWAN python notebook with analytix/spark/hadoop, but it can not find a class:
Py4JJavaError: An error occurred while calling o82.load.
: java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.avro.AvroFileFormat. Please find packages at http://spark.apache.org/third-party-projects.html
I think I passed CMSSpark because it seems to have databriks.spark.avro, too.
I tried it on the command line, but have the same error:
bash-4.2$ spark-submit --packages com.databricks:spark-avro_2.11:4.0.0 test.py 2>&1 | grep Fail
: java.lang.ClassNotFoundException: Failed to find data source: org.apache.spark.sql.avro.AvroFileFormat. Please find packages at http://spark.apache.org/third-party-projects.html
My spark env looks like:
SPARK_HOME=/cvmfs/sft.cern.ch/lcg/releases/spark/2.4.0-cern1-6f44e/x86_64-centos7-gcc7-opt
SPARKMONITOR_KERNEL_PORT=36803
SPARK_CLUSTER_NAME=analytix
SPARK_CONF_DIR=/cvmfs/sft.cern.ch/lcg/etc/hadoop-confext/etc/spark.analytix/conf
SPARK_PORTS=32832,40308,36371,34794,36297,39138
SPARK_USER=bockjoo
HADOOP_TOKEN_FILE_LOCATION=/spark/hadoop.toks
PYSPARK_PYTHON=/cvmfs/sft.cern.ch/lcg/releases/Python/2.7.15-c333c/x86_64-centos7-gcc7-opt/bin/python
PYSPARK_DRIVER_PYTHON=/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/bin/python
SPARK_CONFIG_SCRIPT=/cvmfs/sft.cern.ch/lcg/etc/hadoop-confext/hadoop-setconf.sh
SPARK_LOCAL_IP=172.17.0.5
SPARK_DIST_CLASSPATH=/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/etc/hadoop:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/common/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/common/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/hdfs:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/hdfs/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/hdfs/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/yarn/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/yarn/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/mapreduce/lib/:/cvmfs/sft.cern.ch/lcg/views/LCG_95a/x86_64-centos7-gcc7-opt/share/hadoop/mapreduce/
In [2]:
Do you know how to pass the needed packages in SWAN?
Should I add the classpath for org.apache.spark.sql.avro.AvroFileFormat?
If so, which where is the jar file for that?
from cmsspark.
from cmsspark.
Have you tried not using CMSSpark at all? If you just follow https://hadoop-user-guide.web.cern.ch/hadoop-user-guide/getstart/client_cvmfs.html it works fine for me on lxplus. It should work on SWAN if you include the 'CMSSpark options' checkbox. (you don't need to setup the session in SWAN though)
from cmsspark.
Actually I lied, it works fine on the edge nodes but not lxplus.
from cmsspark.
@vkuznet Thanks Velentine! I will try to set up avro jars following run_spark script.
@nsmith Yes, I tried checking CMSSpark options, but it did not work, i.e., when I checked the avro jars from SWAN, I got this:
!for p in $(echo $SPARK_DIST_CLASSPATH | sed 's#:# #g') ; do echo $p ; done | grep avro
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc7-opt/share/hadoop/common/lib/avro-1.7.4.jar
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Mon/x86_64-centos7-gcc7-opt/share/hadoop/mapreduce/lib/avro-1.7.4.jar
I will see if setting the jar like run_spark will help.
By the way, I wanted to read /project/awg/cms/job-monitoring/avro-snappy/year=2019/month=5/day=1/*.avro to correlate events with /project/monitoring/archive/xrootd/raw/gled/
Just so you know.
from cmsspark.
@bockjoo if you're interested in a high-level comparison of these popularity sources, I can forward you some material.
from cmsspark.
@nsmith Yes, please. Thanks!
from cmsspark.
This works on SWAN Bleeding Edge ( @nsmith I am not sure if you meant this Edge or k8 edge or something else ):
from pyspark import SparkContext, SQLContext, SparkConf
from pyspark.sql import SparkSession
conf = SparkConf().setMaster("local").set("spark.driver.memory", "1g").set("spark.executor.memory", "1g").set("spark.jars.packages","org.apache.spark:spark-avro_2.11:2.4.0")
sc = SparkContext(conf = conf)
spark = SparkSession(sc)
readavro = spark.read.format("avro")
df_job = readavro.load("/project/awg/cms/job-monitoring/avro-snappy/year=2019/month=5/day=1/*.avro")
df_job.printSchema()
from cmsspark.
@nsmith- , @bockjoo , this is what I suspected, even though the usage is trivial as Nick pointed out in his first post, the number of use-cases when you'll look at different data-sources will grow and eventually to accommodate all of them you'll converge on something CMSSpark is aiming to.
I'm not against this simple solution, but you should keep in mind that it will be adopted/modified to every source we have on HDFS. Unfortunately, due to heterogeneous sources/formats/structures there is no simple "solution" and I rather in favor to keep such details hidden from end-users and simplify access to all sources via common framework.
from cmsspark.
Related Issues (19)
- Last day of the month is not processed in bin/cron4dbs_condor HOT 4
- rucio_daily.py error due to wrong default date HOT 1
- Missing job reports in JobMonitoring data? HOT 9
- Typo in date string HOT 1
- sparkexjar always empty HOT 23
- --cvmfs option to run_spark de facto mandatory HOT 2
- Reading Avro files as Dataframes instead of RDD? HOT 5
- Make run_spark compatible with the ithdp-client cluster HOT 1
- Wrong type HOT 4
- Typo in variable name HOT 1
- dbs_events script missing header row in output HOT 2
- Corrupt timestamps in classad csvs HOT 1
- Authentication issues on it-hadoop-client HOT 8
- Bug in run_spark? HOT 1
- Wrong field name in processing era dataframe? HOT 5
- Creating check functions for critic cron jobs HOT 1
- Main bash script for cron jobs and exposing cron metrics to push-gateway HOT 3
- Apply check util functions in cron jobs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cmsspark.