ping-centre
A client for easily collecting events and metrics.
Install
$ npm install ping-centreUsage
Node
npm install ping-centre --saveto get the ping-centre dependencies- Import as follows:
const PingCentre = require("ping-centre");
Firefox Addon
- Run npm run bundle to output bundles to
dist/ - Place the
ping-centre.addon.min.jsbundle in the addon directory - Import as follows:
const PingCentre = require("./ping-centre.addon.min");
Browser
- Run npm run bundle to output bundles to
dist/ - Place the
ping-centre.min.jsbundle in your project directory - Import as follows:
<script type="text/javascript" src="ping-centre.min.js" charset="utf-8"></script> PingCentrewill be available as a global
From there, the following code can be executed in any environment:
// create a ping-centre object
const pc = new PingCentre("some_topic_foo", "some_cient_id_123");
// create the payload
const payload = {
"event_type": "ping_session"
};
// send the payload asynchronously
pc.sendPing(payload);
// validate the payload asynchronously
pc.validate(payload);When testing your app with Ping Centre, your data will be sent to a staging server by default.
To send your data to a production server, set the NODE_ENV environment variable to production.
Overview
Ping-centre consists of three main parts: the clients, the data pipeline, and the dashboard.
The clients are responsible for collecting the events and forwarding them to Onyx - the entrance of the data pipeline. Besides Onyx, the data pipeline employes a Disco cluster to run the ETL jobs, which in turn persist the outcome to AWS Redshift. Through re:dash dashboard, the user can access the data warehouse, slice and dice the datasets via SQL queries.
Behind the scenes, a ping-centre client is simply a wrapper around the HTTP POST request. Therefore, it could be implemented in any programming language. And this repo implements it in Javascript.
Topics
As ease-of-use is the primary goal of the client, the client user does not need to specify the telemetry destination, i.e. the endpoint of the Onyx. Instead, the user just specifies the topic of the payload. In fact, Onyx merely exposes a single endpoint and multiplexes all the topics onto that endpoint. The ETL task runner Infernyx will demultiplex the inputs and process each topic separately.
Schemas
For each topic, the user is going to provide a schema to describe the associated payload. As the reference of table schema in Redshift, this schema could also be used by the ETL jobs to conduct the data extraction, cleaning, and transforming.
We use joi-browser to define the schemas for the Javascript client. By convention, all
schemas are saved in the schemas directory with the same name of the topics. In each schema,
the user specifies following attributes in the schema for each topic:
- Field name
- Field modifiers
- type
- required or optional
- length if applicable
- enum values, e.g. ['click', 'search', 'delete']
- see Joi for more details
- Other ETL requirements are attached as comments if applicable
Here is an example:
const Joi = require("joi-browser");
const schema = Joi.object().keys({
// a required string field with no more than 128 characters
client_id: Joi.string().max(128).required(),
// a required javascript timestamp with milliseconds
received_at: Joi.date().timestamp().required(),
// an required enum string field
event: Joi.any().valid(['add', 'delete', 'search']).required(),
// an optional positive integer field
value: Joi.number().integer().positive().optional(),
}).options({allowUnknown: true}); // allow other non-specified fields
/*
* ETL processing
*
* 1. Truncate the milliseconds of the 'received_at', e.g. 147743323232 -> 147743323
* 2. Rename the 'value' field to 'latency' in the database
* 3. Capitalize the 'event' field
*/
module.exports = schema;
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent