Getting negetive wallclock time in the sparklens UI about sparklens HOT 3 CLOSED

@shahidki31 Thanks for raising this. We will take a look at get back to you.
I know of couple of reasons why this happens. The first one is missing job end events in the event log file. When the job end time is not known, we try to estimate it. If these are bad estimates, we can run into negative job time issues. The second reason is multiple jobs running in parallel. Sparklens computes driver time by subtracting "time spent in jobs" from the total wall clock time. With parallel jobs, it becomes a bit tricky to find out "time spent in jobs". We had made some changes to deal with this problem, but perhaps running is something new here. We will check and get back to you.

from sparklens.

@shahidki31 Parallel jobs were not getting considered at one place in the code, which is why you were getting a negative driver wallclock time. I have fixed it and updated the jar. Please check now, it should work correctly. Thanks for reporting 👍

from sparklens.

Thanks @iamrohit @mayurdb for the replies. Yes, I am running jobs in parallel (TPCDS queries basically). Seems, console output is giving the correct results, only UI has the problem. Will check again. Thank you.

from sparklens.