Deep Dive into Apache Spark 深入研读Spark源码
License: Apache License 2.0
These series of spark study notes will deep dive into Apache Spark source codes, which means there will be advanced topics and readers need to be alreadly familiar with Spark APIs.
错误信息如下：
Error:(9, 14) EventBatch is already defined as object EventBatch
public class EventBatch extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
Error:(9, 14) EventBatch is already defined as class EventBatch
public class EventBatch extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
/Users/mingyuliu/Project/sources/spark/external/flume-sink/target/scala-2.11/src_managed/main/compiled_avro/org/apache/spark/streaming/flume/sink/SparkFlumeProtocol.java
Error:(26, 18) SparkFlumeProtocol is already defined as object SparkFlumeProtocol
public interface SparkFlumeProtocol {
Error:(26, 18) SparkFlumeProtocol is already defined as trait SparkFlumeProtocol
public interface SparkFlumeProtocol {
/Users/mingyuliu/Project/sources/spark/external/flume-sink/target/scala-2.11/src_managed/main/compiled_avro/org/apache/spark/streaming/flume/sink/SparkSinkEvent.java
Error:(9, 14) SparkSinkEvent is already defined as object SparkSinkEvent
public class SparkSinkEvent extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
Error:(9, 14) SparkSinkEvent is already defined as class SparkSinkEvent
public class SparkSinkEvent extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {```/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/bin/java -Dspark.master=local "-javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=60857:/Applications/IntelliJ IDEA.app/Contents/bin" -Dfile.encoding=UTF-8 -classpath /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/cldrdata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/jaccess.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/nashorn.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/jfxswt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/packager.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/lib/tools.jar:/Users/fiona/Desktop/spark-1.6.3/examples/target/scala-2.10/classes:/Users/fiona/.m2/repository/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7-hadoop2.jar:/Users/fiona/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar:/Users/fiona/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.2.0/hadoop-yarn-api-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-annotations/2.2.0/hadoop-annotations-2.2.0.jar:/Users/fiona/.m2/repository/net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar:/Users/fiona/.m2/repository/org/apache/curator/curator-recipes/2.4.0/curator-recipes-2.4.0.jar:/Users/fiona/.m2/repository/org/apache/curator/curator-framework/2.4.0/curator-framework-2.4.0.jar:/Users/fiona/.m2/repository/org/apache/curator/curator-client/2.4.0/curator-client-2.4.0.jar:/Users/fiona/.m2/repository/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar:/Users/fiona/.m2/repository/org/slf4j/slf4j-api/1.7.10/slf4j-api-1.7.10.jar:/Users/fiona/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/fiona/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar:/Users/fiona/.m2/repository/org/xerial/snappy/snappy-java/1.1.2.6/snappy-java-1.1.2.6.jar:/Users/fiona/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/Users/fiona/.m2/repository/commons-net/commons-net/2.2/commons-net-2.2.jar:/Users/fiona/.m2/repository/io/netty/netty/3.8.0.Final/netty-3.8.0.Final.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/Users/fiona/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/Users/fiona/.m2/repository/com/thoughtworks/paranamer/paranamer/2.6/paranamer-2.6.jar:/Users/fiona/.m2/repository/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-column/1.7.0/parquet-column-1.7.0.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-common/1.7.0/parquet-common-1.7.0.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-encoding/1.7.0/parquet-encoding-1.7.0.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-generator/1.7.0/parquet-generator-1.7.0.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-hadoop/1.7.0/parquet-hadoop-1.7.0.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-format/2.3.0-incubating/parquet-format-2.3.0-incubating.jar:/Users/fiona/.m2/repository/org/apache/parquet/parquet-jackson/1.7.0/parquet-jackson-1.7.0.jar:/Users/fiona/.m2/repository/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/Users/fiona/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.7/jaxb-impl-2.2.7.jar:/Users/fiona/.m2/repository/com/sun/xml/bind/jaxb-core/2.2.7/jaxb-core-2.2.7.jar:/Users/fiona/.m2/repository/com/twitter/parquet-hadoop-bundle/1.6.0/parquet-hadoop-bundle-1.6.0.jar:/Users/fiona/.m2/repository/org/spark-project/hive/hive-cli/1.2.1.spark/hive-cli-1.2.1.spark.jar:/Users/fiona/.m2/repository/jline/jline/2.12/jline-2.12.jar:/Users/fiona/.m2/repository/org/spark-project/hive/hive-exec/1.2.1.spark/hive-exec-1.2.1.spark.jar:/Users/fiona/.m2/repository/javolution/javolution/5.5.1/javolution-5.5.1.jar:/Users/fiona/.m2/repository/log4j/apache-log4j-extras/1.2.17/apache-log4j-extras-1.2.17.jar:/Users/fiona/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:/Users/fiona/.m2/repository/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:/Users/fiona/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar:/Users/fiona/.m2/repository/org/antlr/ST4/4.0.4/ST4-4.0.4.jar:/Users/fiona/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/Users/fiona/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/Users/fiona/.m2/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/Users/fiona/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/fiona/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/Users/fiona/.m2/repository/org/spark-project/hive/hive-metastore/1.2.1.spark/hive-metastore-1.2.1.spark.jar:/Users/fiona/.m2/repository/com/jolbox/bonecp/0.8.0.RELEASE/bonecp-0.8.0.RELEASE.jar:/Users/fiona/.m2/repository/org/apache/derby/derby/10.12.1.1/derby-10.12.1.1.jar:/Users/fiona/.m2/repository/org/datanucleus/datanucleus-api-jdo/3.2.6/datanucleus-api-jdo-3.2.6.jar:/Users/fiona/.m2/repository/org/datanucleus/datanucleus-rdbms/3.2.9/datanucleus-rdbms-3.2.9.jar:/Users/fiona/.m2/repository/commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar:/Users/fiona/.m2/repository/commons-dbcp/commons-dbcp/1.4/commons-dbcp-1.4.jar:/Users/fiona/.m2/repository/javax/jdo/jdo-api/3.0.1/jdo-api-3.0.1.jar:/Users/fiona/.m2/repository/javax/transaction/jta/1.1/jta-1.1.jar:/Users/fiona/.m2/repository/org/apache/avro/avro/1.7.7/avro-1.7.7.jar:/Users/fiona/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/fiona/.m2/repository/org/apache/calcite/calcite-avatica/1.2.0-incubating/calcite-avatica-1.2.0-incubating.jar:/Users/fiona/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/fiona/.m2/repository/commons-codec/commons-codec/1.10/commons-codec-1.10.jar:/Users/fiona/.m2/repository/joda-time/joda-time/2.9.3/joda-time-2.9.3.jar:/Users/fiona/.m2/repository/org/jodd/jodd-core/3.5.2/jodd-core-3.5.2.jar:/Users/fiona/.m2/repository/org/datanucleus/datanucleus-core/3.2.10/datanucleus-core-3.2.10.jar:/Users/fiona/Desktop/spark-1.6.3/external/twitter/target/scala-2.10/classes:/Users/fiona/.m2/repository/org/twitter4j/twitter4j-stream/4.0.4/twitter4j-stream-4.0.4.jar:/Users/fiona/.m2/repository/org/twitter4j/twitter4j-core/4.0.4/twitter4j-core-4.0.4.jar:/Users/fiona/Desktop/spark-1.6.3/external/flume/target/scala-2.10/classes:/Users/fiona/Desktop/spark-1.6.3/external/flume-sink/target/scala-2.10/classes:/Users/fiona/.m2/repository/org/apache/flume/flume-ng-core/1.6.0/flume-ng-core-1.6.0.jar:/Users/fiona/.m2/repository/org/apache/flume/flume-ng-configuration/1.6.0/flume-ng-configuration-1.6.0.jar:/Users/fiona/.m2/repository/com/google/code/gson/gson/2.2.2/gson-2.2.2.jar:/Users/fiona/.m2/repository/org/apache/mina/mina-core/2.0.4/mina-core-2.0.4.jar:/Users/fiona/.m2/repository/org/apache/flume/flume-ng-sdk/1.6.0/flume-ng-sdk-1.6.0.jar:/Users/fiona/Desktop/spark-1.6.3/external/mqtt/target/scala-2.10/classes:/Users/fiona/.m2/repository/org/eclipse/paho/org.eclipse.paho.client.mqttv3/1.0.1/org.eclipse.paho.client.mqttv3-1.0.1.jar:/Users/fiona/Desktop/spark-1.6.3/external/zeromq/target/scala-2.10/classes:/Users/fiona/.m2/repository/com/typesafe/akka/akka-zeromq_2.10/2.3.11/akka-zeromq_2.10-2.3.11.jar:/Users/fiona/.m2/repository/org/zeromq/zeromq-scala-binding_2.10/0.0.7/zeromq-scala-binding_2.10-0.0.7.jar:/Users/fiona/.m2/repository/net/java/dev/jna/jna/3.0.9/jna-3.0.9.jar:/Users/fiona/.m2/repository/com/github/jnr/jnr-constants/0.8.2/jnr-constants-0.8.2.jar:/Users/fiona/Desktop/spark-1.6.3/external/kafka/target/scala-2.10/classes:/Users/fiona/.m2/repository/org/apache/kafka/kafka_2.10/0.8.2.1/kafka_2.10-0.8.2.1.jar:/Users/fiona/.m2/repository/org/apache/kafka/kafka-clients/0.8.2.1/kafka-clients-0.8.2.1.jar:/Users/fiona/.m2/repository/com/101tec/zkclient/0.3/zkclient-0.3.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-testing-util/0.98.7-hadoop2/hbase-testing-util-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-common/0.98.7-hadoop2/hbase-common-0.98.7-hadoop2-tests.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-server/0.98.7-hadoop2/hbase-server-0.98.7-hadoop2-tests.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/Users/fiona/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/Users/fiona/.m2/repository/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-hadoop2-compat/0.98.7-hadoop2/hbase-hadoop2-compat-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-hadoop2-compat/0.98.7-hadoop2/hbase-hadoop2-compat-0.98.7-hadoop2-tests.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-common/2.2.0/hadoop-common-2.2.0.jar:/Users/fiona/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/Users/fiona/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar:/Users/fiona/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/fiona/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/fiona/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/Users/fiona/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/fiona/.m2/repository/com/jcraft/jsch/0.1.42/jsch-0.1.42.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-auth/2.2.0/hadoop-auth-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.2.0/hadoop-mapreduce-client-core-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.2.0/hadoop-yarn-common-2.2.0.jar:/Users/fiona/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/Users/fiona/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/Users/fiona/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-grizzly2/1.9/jersey-test-framework-grizzly2-1.9.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-test-framework/jersey-test-framework-core/1.9/jersey-test-framework-core-1.9.jar:/Users/fiona/.m2/repository/javax/servlet/javax.servlet-api/3.0.1/javax.servlet-api-3.0.1.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/Users/fiona/.m2/repository/com/sun/jersey/jersey-grizzly2/1.9/jersey-grizzly2-1.9.jar:/Users/fiona/.m2/repository/org/glassfish/grizzly/grizzly-http/2.1.2/grizzly-http-2.1.2.jar:/Users/fiona/.m2/repository/org/glassfish/grizzly/grizzly-framework/2.1.2/grizzly-framework-2.1.2.jar:/Users/fiona/.m2/repository/org/glassfish/gmbal/gmbal-api-only/3.0.0-b023/gmbal-api-only-3.0.0-b023.jar:/Users/fiona/.m2/repository/org/glassfish/external/management-api/3.0.0-b012/management-api-3.0.0-b012.jar:/Users/fiona/.m2/repository/org/glassfish/grizzly/grizzly-http-server/2.1.2/grizzly-http-server-2.1.2.jar:/Users/fiona/.m2/repository/org/glassfish/grizzly/grizzly-rcm/2.1.2/grizzly-rcm-2.1.2.jar:/Users/fiona/.m2/repository/org/glassfish/grizzly/grizzly-http-servlet/2.1.2/grizzly-http-servlet-2.1.2.jar:/Users/fiona/.m2/repository/org/glassfish/javax.servlet/3.1/javax.servlet-3.1.jar:/Users/fiona/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.9/jersey-guice-1.9.jar:/Users/fiona/.m2/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.2.0/hadoop-mapreduce-client-jobclient-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.2.0/hadoop-mapreduce-client-common-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.2.0/hadoop-yarn-client-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.2.0/hadoop-yarn-server-common-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.2.0/hadoop-mapreduce-client-shuffle-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-yarn-server-nodemanager/2.2.0/hadoop-yarn-server-nodemanager-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.2.0/hadoop-hdfs-2.2.0.jar:/Users/fiona/.m2/repository/commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.2.0/hadoop-hdfs-2.2.0-tests.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-minicluster/2.2.0/hadoop-minicluster-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-common/2.2.0/hadoop-common-2.2.0-tests.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-yarn-server-web-proxy/2.2.0/hadoop-yarn-server-web-proxy-2.2.0.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.2.0/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/Users/fiona/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-hs/2.2.0/hadoop-mapreduce-client-hs-2.2.0.jar:/Users/fiona/.m2/repository/com/github/stephenc/findbugs/findbugs-annotations/1.3.9-1/findbugs-annotations-1.3.9-1.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-protocol/0.98.7-hadoop2/hbase-protocol-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/fiona/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-common/0.98.7-hadoop2/hbase-common-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/fiona/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/fiona/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-client/0.98.7-hadoop2/hbase-client-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/fiona/.m2/repository/org/cloudera/htrace/htrace-core/2.04/htrace-core-2.04.jar:/Users/fiona/.m2/repository/org/jruby/joni/joni/2.1.2/joni-2.1.2.jar:/Users/fiona/.m2/repository/org/jruby/jcodings/jcodings/1.0.8/jcodings-1.0.8.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-server/0.98.7-hadoop2/hbase-server-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-prefix-tree/0.98.7-hadoop2/hbase-prefix-tree-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar:/Users/fiona/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/fiona/.m2/repository/com/github/stephenc/high-scale-lib/high-scale-lib/1.1.1/high-scale-lib-1.1.1.jar:/Users/fiona/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/Users/fiona/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/Users/fiona/.m2/repository/org/mortbay/jetty/jetty-sslengine/6.1.26/jetty-sslengine-6.1.26.jar:/Users/fiona/.m2/repository/org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar:/Users/fiona/.m2/repository/org/mortbay/jetty/jsp-api-2.1/6.1.14/jsp-api-2.1-6.1.14.jar:/Users/fiona/.m2/repository/org/mortbay/jetty/servlet-api-2.5/6.1.14/servlet-api-2.5-6.1.14.jar:/Users/fiona/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar:/Users/fiona/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar:/Users/fiona/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar:/Users/fiona/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar:/Users/fiona/.m2/repository/org/jamon/jamon-runtime/2.3.1/jamon-runtime-2.3.1.jar:/Users/fiona/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/Users/fiona/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/fiona/.m2/repository/org/apache/hbase/hbase-hadoop-compat/0.98.7-hadoop2/hbase-hadoop-compat-0.98.7-hadoop2.jar:/Users/fiona/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/Users/fiona/.m2/repository/com/twitter/algebird-core_2.10/0.9.0/algebird-core_2.10-0.9.0.jar:/Users/fiona/.m2/repository/com/googlecode/javaewah/JavaEWAH/0.6.6/JavaEWAH-0.6.6.jar:/Users/fiona/.m2/repository/org/apache/cassandra/cassandra-all/1.2.6/cassandra-all-1.2.6.jar:/Users/fiona/.m2/repository/org/antlr/antlr/3.2/antlr-3.2.jar:/Users/fiona/.m2/repository/com/googlecode/json-simple/json-simple/1.1/json-simple-1.1.jar:/Users/fiona/.m2/repository/org/yaml/snakeyaml/1.6/snakeyaml-1.6.jar:/Users/fiona/.m2/repository/edu/stanford/ppl/snaptree/0.1/snaptree-0.1.jar:/Users/fiona/.m2/repository/org/mindrot/jbcrypt/0.3m/jbcrypt-0.3m.jar:/Users/fiona/.m2/repository/org/apache/cassandra/cassandra-thrift/1.2.6/cassandra-thrift-1.2.6.jar:/Users/fiona/.m2/repository/com/github/stephenc/jamm/0.2.5/jamm-0.2.5.jar:/Users/fiona/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar:/Users/fiona/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/fiona/Desktop/spark-1.6.3/assembly/target/scala-2.10/spark-assembly-1.6.3-hadoop2.7.2.jar org.apache.spark.examples.LogQuery
objc[10774]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/bin/java (0x1043c74c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_92.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10448f4e0). One of the two will be used. Which one is undefined.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/fiona/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/fiona/Desktop/spark-1.6.3/assembly/target/scala-2.10/spark-assembly-1.6.3-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-06-27 23:33:18,731 INFO [main] spark.SparkContext (Logging.scala:logInfo(58)) - Running Spark version 1.6.3
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/collect/Maps
at org.apache.hadoop.metrics2.lib.MetricsRegistry.(MetricsRegistry.java:42)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.(MetricsSystemImpl.java:87)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.(MetricsSystemImpl.java:133)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.(DefaultMetricsSystem.java:38)
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.(DefaultMetricsSystem.java:36)
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:99)
at org.apache.hadoop.security.UserGroupInformation.(UserGroupInformation.java:192)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2214)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2214)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2214)
at org.apache.spark.SparkContext.(SparkContext.scala:322)
at org.apache.spark.examples.LogQuery$.main(LogQuery.scala:48)
at org.apache.spark.examples.LogQuery.main(LogQuery.scala)
Caused by: java.lang.ClassNotFoundException: com.google.common.collect.Maps
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
Process finished with exit code 1
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/immutable/List
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: scala.collection.immutable.List
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
麻烦帮我看看这个错
我在添加jars之前都和题主一样。但是添加完jars之后出现了以下错误，能不能帮我看看哪里出了问题
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/08/02 16:56:21 INFO SparkContext: Running Spark version 2.3.0-SNAPSHOT
17/08/02 16:56:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/02 16:56:23 WARN Utils: Your hostname, hjw-XiaoXin-700 resolves to a loopback address: 127.0.1.1; using 192.168.191.4 instead (on interface wlp2s0)
17/08/02 16:56:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
17/08/02 16:56:23 ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.(SparkContext.scala:368)
at org.apache.spark.examples.LogQuery$.main(LogQuery.scala:47)
at org.apache.spark.examples.LogQuery.main(LogQuery.scala)
17/08/02 16:56:23 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$postApplicationEnd(SparkContext.scala:2401)
at org.apache.spark.SparkContext$$anonfun$stop$1.apply$mcV$sp(SparkContext.scala:1890)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1331)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1889)
at org.apache.spark.SparkContext.(SparkContext.scala:581)
at org.apache.spark.examples.LogQuery$.main(LogQuery.scala:47)
at org.apache.spark.examples.LogQuery.main(LogQuery.scala)
17/08/02 16:56:23 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.(SparkContext.scala:368)
at org.apache.spark.examples.LogQuery$.main(LogQuery.scala:47)
at org.apache.spark.examples.LogQuery.main(LogQuery.scala)
应该是java版本问题，但是我确定本地是jdk 1.8+
您好。sequenceFile返回的RDD不能正常的persist，我想重写这个功能（比如sequenceFilePersist），最终目标是让返回RDD可以不通过map函数直接persist。想请问，我需要从继承或者扩展哪个类下手呢？十分感谢。
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.