Flight Recorder Mode when Driver crashed from OOM about sparkmeasure HOT 3 CLOSED

If your goal is to investigate OOMs in the Spark driver, sparkMeasure is not the best tool.
Spark metrics instrumentation has metrics for Java memory usage on the driver (and executors), which may be useful for your case, see https://github.com/LucaCanali/Miscellaneous/tree/master/Spark_Dashboard
Or else you can try using JAVA instrumentation like JFR, or debugging tools like https://github.com/jvm-profiling-tools/async-profiler

from sparkmeasure.

Hi,

Flight recorder mode with file output currently has a very simple implementation, where all metrics are buffered into the driver memory and will only be written out as the application finishes, which indeed is a problem if youhave a driver crash in between.

You may want to have a looks the Filght recorder mode with InfluxDB sink and/or Kafka sink, to overcome those issues: https://github.com/LucaCanali/sparkMeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md https://github.com/LucaCanali/sparkMeasure/blob/master/docs/Flight_recorder_mode_KafkaSink.md

You may also want to check the SPark dashboard project https://github.com/cerndb/spark-dashboard

Best,
Luca

from sparkmeasure.

@LucaCanali
Thanks for the reply. Even if Filght recorder is using InfluxDB sink and/or Kafka sink, it would not be able to report metrics of the latest / on-going task/stage/executor at the time of the crash right? One solution might be to have some sort of monitoring done outside of Spark Driver? Do you know of anything like that or is there any other workaround I might have missed? Thanks!

from sparkmeasure.