Tests using Scala and Spark for Big Data projects.
An article was published explaining:
- https://edersoncorbari.github.io/tutorials/scala-spark-graph/
- https://dzone.com/articles/bigdata-developing-a-graph-in-spark-and-scala
- https://dzone.com/articles/hdfs-offline-analysis-of-fsimage-metadata
- The template of a project using SBT;
- A top-down graph to find a company's employee hierarchy;
- Parsing XML file: FsImage (HDFS).
To compile the project, you must have the following tools installed:
- SBT-1.2.0 or greater.
Now run the commands below to compile the project:
$ git clone https://github.com/edersoncorbari/scala-lab.git
$ cd scala-lab
$ sbt update compile run
If you receive an error of type: "Caused by: java.net.UnknownHostException: your-host: localhost: Name or service not known", your hostname is not configured in the /etc/hosts please add and run again.
To perform all the tests please run the command:
$ sbt test
To run a specific test run the command:
$ sbt
Within the SBT console perform a specific test:
> testOnly io.github.edersoncorbari.graph.HierarchyEmployeeTest
> testOnly io.github.edersoncorbari.hdfs.FsImageTest