rathboma / hadoop-framework-examples Goto Github PK

An implementation of a real-world map-reduce workflow in each major framework.

License: Other

Java 65.90% Scala 24.28% Shell 2.27% Python 6.57% PigLatin 0.97%

hadoop-framework-examples's Introduction

Realistic Hadoop Data Processing Examples

This code is to accompany my blog post on map reduce frameworks

The point of the code in this repository is to provide an implementation for a business question (listed below) in each of the major Map Reduce frameworks.

Each implementation will get it's own subdirectory with it's own build and running instructions. Each framework will also get an accompanying test, and an in-depth walkthrough about implementation details.

The following implementations are complete:

Java map reduce - walkthrough
Scoobi - walkthrough
Scalding - walkthrough
Cascading - walkthrough
Hive - walkthrough
Pig - walkthrough

The problem

The Data

We have two datasets: customers, and transactions.

Customer Fields:

id (1)
email ([email protected])
language (EN)
location (US)

Transaction Fields:

transaction-id (1)
product-id (1)
user-id (1)
purchase-amount (19.99)
product-description (a rubber chicken)

These two datasets are stored in tab-delimited files somewhere on HDFS.

The Question

For each product, we want to know the number of locations in which that product was purchased.

That's it!

In the real world, we might have other questions, like the number of purchases per location for each product.

hadoop-framework-examples's People

Contributors

Stargazers

Watchers

Forkers

yubingzhong bhardwaj01 shashankl harshabhat86 wqren heiz33 glee314 amimimor kwen-adchemix chinna1986 rameshvinbox ritchie-xl kirankumaralugonda vinodreddyb bhanodai whossa myang321 asitatikie raveendrau mycherrybox arpit-vizury jradd mengyou658 amoghd33 gantzmachine helenahm sambhav13 cocoon24 mutyalart hadoop-examples edogdu kdwcse lingswang bvkkr2808 sensaid srinathji mdsb szz27106 anlenhk himanshu4141 redskycry am014si bsalimi ssrisunt qmgeng iomsingh pvk2727 goutam-git vaquarkhan gokcerbelgusen sailendramanne narayantalk2u jpmanne kabachaa mahendrakapadne jaijujacob bindas45 bhaskar85930 dataedgesystems dataedgesystem hjkcc appsynergy himanshubhardwaj sb444 subh1801 pratitidevelop tonnydp nagendra176 clamus seanmoynihan anjijava16 wentium andreyzhadan anirudhreddy92 smahesh17 yhwen julia844 chandangs16 ramzisolt galleria ry1031 deysubhasish zedfauji ram1991 jnarvaezp harishrajoori bigdata06 code-o-swagger albertofbs15 hiroshigawa dexter1618 rasikboy kvasagiri vijaykumar243 xtutran yatharthtripathi albertoesteva88 jevenzh abdul99 dmarinav

hadoop-framework-examples's Issues

new issue

Run the Apache Spark Pi job in yarn-client mode, using code from org.apache.spark:
./bin/spark-submit --class org.apache.spark.examples.SparkPi
--master yarn-client
--num-executors 1
--driver-memory 512m
--executor-memory 512m \

Issue with Building Jar using Maven and Executing Application

Hey Matthew,
I'm having some difficulty with running the "Spark-Scala" application and was following your blog post;. I noticed that you skipped the jar packaging step in the guide, so I followed the Github repository readme by running "mvn compile". However, this doesn't create the jar file within the "target" directory; instead, the "target" directory has a directory "classes" and a classes.timestamp file, and "classes" has the "ExampleJob.class" bytecode. Could you assist with this issue?

java.lang.ClassNotFoundException: main.scala.com.matthewrathbone.spark.Main

Error: java.io.IOException: wrong key class: hadoop_join.TextTuple is not class org.apache.hadoop.io.Text

I copy your hadoop join example and run , but it raise error, I can not figure it out. following is some logs, and my env is hadoop 2.7.5

18/01/19 22:54:28 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.1.13:8032
18/01/19 22:54:29 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/01/19 22:54:29 INFO input.FileInputFormat: Total input paths to process : 1
18/01/19 22:54:29 INFO input.FileInputFormat: Total input paths to process : 1
18/01/19 22:54:29 INFO mapreduce.JobSubmitter: number of splits:2
18/01/19 22:54:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1516287880873_0006
18/01/19 22:54:30 INFO impl.YarnClientImpl: Submitted application application_1516287880873_0006
18/01/19 22:54:30 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1516287880873_0006/
18/01/19 22:54:30 INFO mapreduce.Job: Running job: job_1516287880873_0006
18/01/19 22:54:38 INFO mapreduce.Job: Job job_1516287880873_0006 running in uber mode : false
18/01/19 22:54:38 INFO mapreduce.Job:  map 0% reduce 0%
18/01/19 22:54:47 INFO mapreduce.Job:  map 100% reduce 0%
18/01/19 22:54:52 INFO mapreduce.Job: Task Id : attempt_1516287880873_0006_r_000000_0, Status : FAILED
Error: java.io.IOException: wrong key class: hadoop_join.TextTuple is not class org.apache.hadoop.io.Text
	at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1375)
	at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83)
	at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
	at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:150)
	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
	at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

rathboma / hadoop-framework-examples Goto Github PK

hadoop-framework-examples's Introduction

Realistic Hadoop Data Processing Examples

The problem

The Data

The Question

hadoop-framework-examples's People

Contributors

Stargazers

Watchers

Forkers

hadoop-framework-examples's Issues

new issue

Issue with Building Jar using Maven and Executing Application

java.lang.ClassNotFoundException: main.scala.com.matthewrathbone.spark.Main

Error: java.io.IOException: wrong key class: hadoop_join.TextTuple is not class org.apache.hadoop.io.Text

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent