No Module in PySpark about hadoop-pcap HOT 4 CLOSED

What sort of error or log messages do you get when it fails?

from hadoop-pcap.

Sorry, I thought I had attached the error messages:

Using Python version 2.7.12 (default, Nov 19 2016 06:48:10)
SparkSession available as 'spark'.

import net.ripe.hadoop.pcap.io.PcapInputFormat
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named net.ripe.hadoop.pcap.io.PcapInputFormat
import net.ripe.hadoop.pcap.io.CombinePcapInputFormat
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named net.ripe.hadoop.pcap.io.CombinePcapInputFormat

from net.ripe.hadoop.pcap.io import CombinePcapInputFormat
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named net.ripe.hadoop.pcap.io

I have started the pyspark with this options:

$SPARK_HOME/bin/pyspark --master spark://base:7077 --executor-memory 1000m --executor-cores 4 --conf "spark.locality.wait.node=0" --conf "spark.executor.extraJavaOptions=-XX:MaxDirectMemorySize=1000m" --conf "spark.default.parallelism=3" --driver-memory=600m --jars $SPARK_HOME/mysql-connector-java-5.1.41-bin.jar,$SPARK_HOME/hadoop-pcap-serde-1.1-jar-with-dependencies.jar,$SPARK_HOME/hadoop-pcap-lib-1.1.jar --verbose --num-executors 10 --driver-class-path=/usr/local/spark/*

using the same cmd line in spark-shell works good.

from hadoop-pcap.

That won't work. hadoop-pcap is a Java library. So you will have to figure out how to use it with something like Py4J or similar. Haven't done that myself yet so cannot help on how.

from hadoop-pcap.

I have tried using py4j. It imports the library.
The real issue is when I pass the class to the newAPIhadoopFile. It throws a classnotFound.

from hadoop-pcap.