In this service, lots and lots of things are RDDs.

Document it, put it in the hello world script?

Links to: https://github.com/mozilla/telemetry-server/tree/master/http/analysis-service

Suggestions for 'right patterns' for all the various transforms are also helpful.

I'm currently working on a clean-slate implementation of Telemetry as a self-contained library, for Servo, and I need a few clarifications regarding the format accepted for histograms. See this issue for this specific piece of work.

Fields that make no sense for some histograms

If I read the source code correctly, convert.py will set sum, sum_squares_lo, sum_squares_hi, log_sum, log_sum_squares to -1 if these fields cannot be found. Does this mean that histograms that have no meaningful values for either of these fields (e.g. enumerated histograms, count histograms, boolean histograms, flag histograms) can omit these fields? Will e.g. the dashboard still work?

Min/max

For count histograms, if I read correctly the C++ source, the min is harcoded to 1, the max is hardcoded to 2 and the number of buckets is hardcoded to 3. Can I deduce that all three are ignored?

(I may have other questions later)

We should add a swap file to the self-service telemetry instances to avoid an out of memory error in some cases.

I found myself needing to perform data dedeplication for the update hotfix data. (Yes, I know incremental upload works better with map reduce.)

My ideal deduplication mechanism is:

1. Collect the upload date of all records with the same ID
2. Throw away all but the most recent record

i.e. last write wins

It is difficult to do this in telemetry-server today if N>1 mappers are involved because the mapper receives no metadata about upload time. Call this a deficiency in how the hotfix data stream is constructed if you want. You can do this with N=1 mappers, but it is absurdly slow (I have multiple cores and I want to use them, dammit). You can write out the original records in map() and have the reducer produce essentially new streams from the filtered output. But that involves tons of extra CPU and I/O. Why should I rewrite a multi-gigabyte data set where the duplication rate is low?

I propose adding the source file/offset information into map(). In my dedupe job, I can write the record ID and file/offset information. A reducer can then find IDs with multiple records and produce an output listing either the "good" or "bad" sets of file/offsets. I can then load this file into another MR job and filter incoming records against it. You pay a penalty for hash lookup on each record, but that should be fast if set is used.

I've already hacked up telemetry-server to pass this extra information to map(). But it breaks the API of map(). Next step is likely to hook up inspect.getargspec() to see if the callee supports the new arguments and pass them if it does. But I wanted to get feedback from people before I fully implement this, as the changes are probably a bit controversial.

Currently, the rsync / scp of the files is a drag. There are lots of solutions here, including autosync with s3, or gist or github.

I have been thinking something along the lines of this:

## GREGG TOTALLY STINKS AT BASH, this needs to be... sourceable?

export -f TELEMETRYCLUSTER=ec2-54-201-192-120.us-west-2.compute.amazonaws.com;
export TELDIR=~/telemetry-analysis-files
mkdir -p "${TELDIR}"

echo -n "${TELEMETRYCLUSTER}" > ~/.telemtry-cluster
echo -n "${TELDIR}" > ~/.telemetry-dir


# connect
tel-connect () { ssh -i my-private-key -L 4040:localhost:4040 -L 8888:localhost:8888 hadoop@`cat ~/.cluster`; }

# scp down
tel-download () { scp -i my-private-key hadoop@$(cat ~/.cluster):~/analyses/* "${TELDIR}"; }

# scp up
tel-upload () { scp -i my-private-key "${TELDIR}"/* hadoop@$(cat ~/.cluster):~/analyses/ ; }

export tel-connect
export tel-upload
export tel-download

Maybe suggesting putting this is some snippet you can cram into a .bash_profile or whatnot :)

Various MR jobs are using the built-in json or simplejson packages for deserializing json payloads. Switching to ujson gives a significant speed-up.

I have a single day of the Firefox update hotfix payloads cached locally. There are 627,404 records that lz4 decompress to 11,655,091,683 bytes. I have a dead simple MR script that performs a JSON deserialize and extracts a single value from the payload and combines it.

Here is the performance of that job with 8 concurrent processes on 4+4HT cores with various JSON implementations.

built-in json

real 0m54.250s
user 6m26.780s
sys 0m9.834s

real 0m53.691s
user 6m22.161s
sys 0m9.710s

real 0m52.698s
user 6m14.038s
sys 0m9.596s

simplejson

real 0m34.825s
user 4m7.692s
sys 0m7.125s

real 0m34.218s
user 4m3.766s
sys 0m7.055s

real 0m34.830s
user 4m4.105s
sys 0m7.043s

ujson

real 0m26.212s
user 3m6.775s
sys 0m5.789s

real 0m27.636s
user 3m16.358s
sys 0m6.077s

real 0m28.094s
user 3m18.188s
sys 0m6.227s

Averages

The averages for CPU time is:

json: 391s
simplejson: 252s
ujson: 200s

lzma --decompress --stdout on this data set takes about 83s of CPU time.

As the data demonstrates, ujson is significantly faster than simplejson and will thus make Telemetry jobs faster and more efficient.

My data should not need validation: any Google search on "Python json benchmark" will tell you others have reached the same conclusion that ujson is the bomb.

It would be great if we could upgrade to Ubuntu 14 as it comes with a newer R version which has better compatibility with some useful packages like dplyr.

Per irc today :) Maybe put a snippit / git link near the page with the on-going spark job, or wiki this up a bit?

GL

07:13 < gregglind> peeps, what's the recommended path for using a spark job as the basis for a dashboard?
07:14 < gregglind> 1.  run it as a cron... then get the data from S3?  Is there permissions and stuff to deal with?
07:14 < gregglind> (i.e., wanting to setup a json or csv that other inside-moz dashboards can use)
07:15 <@mreid> gregglind: public dashboards are easy.  moz-internal dashboards take a little more effort, but it's doable.
07:20 < gregglind> tell me how to make the easy :)
07:27 <@mreid> gregglind: run a scheduled spark job, set it to public, and write out the data you want for your dashboard to
               $PWD (or just make the notebook itself be the dashboard)
07:27 <@mreid> gregglind: then your output will be web-accessible at a URL like this:
07:27 <@mreid> https://analysis-output.telemetry.mozilla.org/mreid-test-new-dev/data/test.txt
07:30  * gregglind confused, doesn't the machine die?
07:30 <@mreid> gregglind: if you want to just have a notebook that updates (with embedded viz and stuff), it would appear at
               https://analysis-output.telemetry.mozilla.org/<job_name>/data/MyAwesomeNotebook.ipynb
07:30 <@mreid> gregglind: it's magic, don't worry :P
07:31 < gregglind> I am trying to avoid 'the notebook is the dashboard.
07:31 < gregglind> Okay :)
               is $PWD) get auto-uploaded to S3 for posterity
07:31 < gregglind> Can you maybe just copy that to a little gist :) I will file a bug against telemetry dash :)
07:32 <@mreid> so you could also output "my_fancy_dataset.csv[.gz]" and it would become available via a similar URL

There is at least one Python package that needs to be installed before running telemetry:

boto.s3.connection (pip install boto) -- The map/reduce driver 'job.py' could possibly be modified to skip importing this when it's used --local-only

Nice code! Unfortunately I don't see any licensing info anywhere :(

I can haz license boilerplate in the source files?

Right now, the cluster monitor page is sufficient, but not awesome.

Suggestion... have some better code snippets to handle upload, download and connect.

I know the eventual route is to go auto gist / s3 / whatever. Maybe this is a temp solution?

# remember that gregg is the worst at bash!
export TELEMETRYCLUSTER=ec2-54-201-192-120.us-west-2.compute.amazonaws.com; 
export TELDIR=~/telemetry-analysis-files
mkdir -p "${TELDIR}"

echo -n "${TELEMETRYCLUSTER}" > ~/.telemetry-cluster
echo -n "${TELDIR}" > ~/.telemetry-dir

# connect
tel-connect () { ssh -i my-private-key -L 4040:localhost:4040 -L 8888:localhost:8888 hadoop@`cat ~/.telemetry-cluster`; }

# scp down
tel-download () { scp -i my-private-key hadoop@$(cat ~/.telemetry-cluster):~/analyses/* "$(cat ~/.telemetry-dir)"; }

# scp up
tel-upload () { scp -i my-private-key "$(cat ~/.telemetry-dir)"/* hadoop@$(cat ~/.telemetry-cluster):~/analyses/ ; }

export -f tel-connect
export -f tel-upload
export -f tel-download

In notify.py Allow multiple --attach some_file.txt command line arguments, and add each one as an attachment to the email.

See example code here: https://gist.github.com/yosemitebandit/2883593