qf0188 / googleclusterdata Goto Github PK

##2014-11-18 Report

• Runtime of Jobs/Tasks
• Machine Avaiblility
  • Attributes are obfusicated: We might need to makes some educated guesses
  • Failure rates: This can help us in failure prediction/modelling
• Task Requests: Using the task resource requests and machine availability, we can check whether over provisioning is actually true.

###Challenges

• Data is 217GB. We have still not run into any problems but we would like to prepare ourselves.
• Plan to tranfer the tables that we build into a DB so that join operations are easy

###TODO:

• Two more tables to look at
• Start looking at publications to see if they produce some result. This would help us validating our results.

##2014-11-25 Report ##Pending from next week Last weeks TODO

##Work Done

• Find the number of jobs running a given time instant
• Idea was to find if there is any periodicity in the data
• Plotted busy figures. Looks like there are some patterns.

##Ideas

• Just take a slice of a day to see atleast daily periodicity
• Should I try smoothing?

##Challenges

• Ran out of memory. Tried subsampling and it worked! (Any other ideas?)
• matplotlib is not a good tool to plot graphs with large number of data points
• gnuplot was the next best option for me. Even she revolted!
• Again subsampling! sed is a good tool to use. (I think "El Presidente" mentioned it during his presentation!)
• Used pypy. I would encourage all to use it...it's fast!!

##TODO

• Good news...we have more data. Google just released a new column of the data. They claim "The new data is a randomly-picked 1 second sample of CPU usage from within the associated 5-minute usage-reporting period for that task. Using this data, it will be possible to build up a stochastic model of task utilization over time for long-running tasks."