Repository with implementation to all 6 modules for final capstone (https://www.coursera.org/learn/scala-capstone). Some tips and code snippets are shown here.
Implemented with Spark. I wouldn't recommend trying to implement any other module with Spark. Next modules require highly efficient implementations (a few milliseconds per iteration) to pass the grader tests. Spark creates too much overhead for non-distributed data. Also, be very careful to follow all the requirements when preparing both data sets and specially in your JOIN logic (null keys should also be compared):
val joined = stations.joinWith(temperatures,
stations("stnId") <=> temperatures("stnId") &&
stations("wbanId") <=> temperatures("wbanId")
)
/*
Alternative using string literals:
import org.apache.spark.sql.Column
import org.apache.spark.sql.functions._
def compareColumns(c: String): Column =
coalesce(stations(c), lit("*")) === coalesce(temperatures(c), lit("*"))
val joined = stations.joinWith(temperatures,
compareColumns("stnId") && compareColumns("wbanId")
)
*/Probably the hardest module. You need to implement an algorithm to interpolate Temperatures for every single Location in the map and it has to be FAST; you need to reduce your time to a few (in single digits) milliseconds per temperature interpolated or the grader will throw an annoying Timeout error. Most people suggest (in the forums) to use the closes data points only, not all data points in a single year (using "par" is good enough as long as your implementation is correct). To reduce the number of data points you can use Java quickSort (fastest way to sort by far) and take. My implementation (without helper methods) is:
def predictTemperature(temperatures: Iterable[(Location, Temperature)], location: Location):
Temperature = {
// calculate distances
val distanceTempPairs: ParIterable[(Double, Temperature)] =
temperatures.par.map {
case (l: Location, t: Temperature) => (getDistance(location, l), t)
}
/*
Optional implementation using quicksort and fewer distances:
val arrTest = distanceTempPairs.toArray
scala.util.Sorting.quickSort(arrTest)
val arrTest2 = arrTest.take(40)
*/
// No interpolation needed if distance < 1km.
val closeP = distanceTempPairs.filter(_._1 < 1).take(1)
if (closeP.nonEmpty) closeP.head._2
// else implement the Inverse Distance Weighting formula
else getInverseDistanceWeight(distanceTempPairs)
}I am using all distances and still passing all test by just using parallel computation.
Not too difficult once you understand how to generate Location coordinates from Tile coordinates.
For this module I use memoization at the Grid level only. I use the following helper method as a wrapper:
def memoizeFnc[K, V](dict: MMAP[K, V])(f: K => V): K => V = {
k =>
dict.getOrElse(k, {
dict.update(k, f(k))
dict(k)
})
}This wrapper is used together with a local Map to cache all computed temperatures:
val GRID_MAPS: MMAP[Int, MMAP[GridLocation, Temperature]] = MMAP.empty
/**
* @param temperatures Known temperatures
* @return A function that, given a latitude in [-89, 90] and a longitude in [-180, 179],
* returns the predicted temperature at this location
*/
def makeGrid(temperatures: Iterable[(Location, Temperature)]):
GridLocation => Temperature = {
val tempsHash = temperatures.hashCode()
if (!GRID_MAPS.contains(tempsHash))
GRID_MAPS.update(tempsHash, MMAP.empty[GridLocation, Temperature])
val memoizedGetTemperature = memoizeFnc(GRID_MAPS(tempsHash))(
(gridLoc: GridLocation) => predictTemperature(temperatures, gridLoc.toLocation)
)
memoizedGetTemperature
}You can use this approach or you could just pre-compute all temperatures.
The hardest part is the conversion of Location to CellPoint. Other than that the implementation is similar to Interaction. My approach in visualizeGrid:
val allColors: Array[Color] = allLocations.par
.map { loc =>
val lat = loc.lat.toInt
val lon = loc.lon.toInt
bilinearInterpolation(
CellPoint(loc.lon - lon, loc.lat - lat),
grid(GridLocation(lat, lon)),
grid(GridLocation(lat + 1, lon)),
grid(GridLocation(lat, lon + 1)),
grid(GridLocation(lat + 1, lon + 1)))
}
.map(temperature => interpolateColor(colors, temperature))
.toArrayImplemented using Reactive Programming. Probably the easiest module.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent