Project from the CTU Big Data course which purpose was to compute tf-idf values for the czech wikipedia
Basic ideas behind the tf-idf implementation you can read on my blog.
Input to the mapreduce program should be raw documents where each of them is represented as a line
text text text tex
txt text word
.
.
.
xword yword
Output of this program will be the final TF-IDF matrix:
doc1 car:0.9 plane:1.6 computer:2.3
doc2 elephant:0.2 hadoop:1.1
.
.
.
docn mahout:0.8 storm:1.6
Usage:
1) You need to create jar: use command: mvn clean package
2) Copy the jar to the machine where you have your Hadoop installed
3) Run program: hadoop jar [name of the jar] com.zikesjan.bigdata.TfIdfMain [input path] [output path]