betl's Introduction

BETL

BETL (say: beetle) makes it easy to define Basic ETL jobs for Firefox Telemetry data.

This project is in early alpha.

For many jobs, we only want to do the following:

filter to a subset of telemetry pings
flatten the JSON ping to a flat table
format or perform simple transformations over the field

BETL allows you to define these ETL jobs in a declarative format, removing most of the complexity incurred while transforming pings.

Starting with a collection of Pings, we can define a transformation:

transformed_dataframe = convert_pings(
        sqlContext,
        collection_of_pings,
        DataFrameConfig(
            [
                ("new_column_name", "ping_field", transformation, pyspark.sql.FieldType()),
                ("channel", "meta/normalizedChannel", None, StringType()),
                ("is_release", "meta/normalizedChannel", lambda x: x == "release", BooleanType()),
                ...
            ],
            lambda ping: ping['payload/test'] == '@testpilot-addon'
        )
)

Take a look at this simple example, or this more complex ETL Job

BETL is written in python, so it's easy to get started. We also provide a simple Jupyter notebook, which allows you to schedule your job in ATMO while keeping your code in a version controlled python library.