Goldrush is a small Erlang app that provides fast event stream processing

• per module private event processing statistics
• query module logic can be combined for any/all filters
• query module logic can be reduced to efficiently match event processing

• match input events with greater than (gt) logic
• match input events with less than (lt) logic
• match input events with equal to (eq) logic
• match input events with wildcard (wc) logic
• match input events with notfound (nf) logic
• match no input events (null blackhole) logic
• match all input events (null passthrough) logic

• Once a query has been composed the output action can be overriden with one or more erlang functions. The functions will be applied to each output event from the query.

• Values stored are also provided to functions called on event output.
• Handle job execution and timing which can also get values stored
• create input events that include runtime on successful function executions.

• provide state storage option to compile, caching the values in query module.

To use goldrush in your application, you need to define it as a rebar dep or include it in erlang’s path.

Before composing modules, you’ll need to define a query. The query syntax matches any number of `{erlang, terms}’ and is composed as follows:

• Simple logic is defined as any logic matching a single event filter

Select all events where ‘a’ exists and is greater than 0.

glc:gt(a, 0).

Select all events where ‘a’ exists and is greater than or equal to 0.

glc:gte(a, 0).

Select all events where ‘a’ exists and is equal to 0.

glc:eq(a, 0).

Select all events where ‘a’ exists and is not equal to 0.

glc:neq(a, 0).

Select all events where ‘a’ exists and is less than 0.

glc:lt(a, 0).

Select all events where ‘a’ exists and is less than or equal to 0.

glc:lte(a, 0).

Select all events where ‘a’ exists.

glc:wc(a).

Select all events where ‘a’ does not exist.

glc:nf(a).

Select no input events. User as a black hole query.

glc:null(false).

Select all input events. Used as a passthrough query.

glc:null(true).

• Combined logic is defined as logic matching multiple event filters

Select all events where both ‘a’ AND ‘b’ exists and are greater than 0.

glc:all([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ OR ‘b’ exists and are greater than 0.

glc:any([glc:gt(a, 0), glc:gt(b, 0)]).

Select all events where ‘a’ AND ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:all([glc:gt(a, 1), glc:lt(b, 2)]).

Select all events where ‘a’ OR ‘b’ exists where ‘a’ is greater than 1 and ‘b’ is less than 2.

glc:any([glc:gt(a, 1), glc:lt(b, 2)]).

• Reduced logic is defined as logic which can be simplified to improve efficiency.

Select all events where ‘a’ is equal to 1, ‘b’ is equal to 2 and ‘c’ is equal to 3 and collapse any duplicate logic.

glc_lib:reduce(
    glc:all([
        glc:any([glc:eq(a, 1), glc:eq(b, 2)]),
        glc:any([glc:eq(a, 1), glc:eq(c, 3)])])).

The previous example will produce and is equivalent to:

glc:all([glc:eq(a, 1), glc:eq(b, 2), glc:eq(c, 3)]).

• All query modules must be compiled before use

To compose a module you will take your Query defined above and compile it.

glc:compile(Module, Query).
glc:compile(Module, Query, State).
glc:compile(Module, Query, State, ResetStatistics).

• At this point you will be able to handle an event using a compiled query.

Begin by constructing an event list.

Event = gre:make([{'a', 2}], [list]).

Now pass it to your query module to be handled.

glc:handle(Module, Event).

• You can override the output action with an erlang function.

Write all input events as info reports to the error logger.

glc:with(glc:null(true), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is less than 5 as info reports to the error logger.

glc:with(glc:lt(error_level, 5), fun(E) ->
     error_logger:info_report(gre:pairs(E)) end).

Write all input events where `error_level’ exists and is 3 or 5 as info reports to the error logger.

glc:any([
  glc:with(glc:lt(error_level, 3), fun(E) ->
      error_logger:info_report(gre:pairs(E)) end),
  glc:with(glc:lt(error_level, 5), fun(E) ->
      error_logger:info_report(gre:pairs(E)) end)]).

To compose a module with state data you will add a third argument (orddict).

glc:compile(Module, Query, [{stored, value}]).

Return the stored value in this query module.

{ok, value} = glc:get(stored).

Return all stored values in this query module.

[...] = Module:get().

• You can create query modules with local state to compare to event data in `with’ and `run’

To compose a module with state data you will add a third argument (orddict).

glc:compile(Module, Query, [{stored, value}]).

• You can use query modules in a way similar to mochiglobal

Return the stored value in this query module.

{ok, value} = glc:get(stored).

• You can use query modules to execute jobs, if the job errors or not, process an event.
• `with’ is similar to `run’, the main difference is additional statistics and execution order
• when a job completes in error, the event data will contain an additional {error, _} item

To execute a job through the query module, inputting an event on success.

Event = gre:make([{'a', 2}], [list]).
{ExecutionTime, Result}= glc:run(Module, fun(Event, State) ->
    %% do not end with {error, _} or throw an exception 
end, Event).

Return the number of input events for this query module.

glc:input(Module).

Return the number of output events for this query module.

glc:output(Module).

Return the number of filtered events for this query module.

glc:filter(Module).

Return the number of job runs for this query module.

glc:job_run(Module).

Return the number of job errors for this query module.

glc:job_error(Module).

Return the number of job inputs for this query module.

glc:job_input(Module).

Return the amount of time jobs took for this query module.

glc:job_time(Module).

• This is really just a drop in the bucket.

Return the average time jobs took for this query module.

glc:job_time(Module) / glc:job_input(Module) / 1000000.

Return the query combining the conditional logic of multiple modules

glc_lib:reduce(glc:all([Module1:info('query'), Module2:info('query')]).

Return all statistics from this query module.

glc:info(Module).

$ ./rebar compile

or

$ make

0.1.9

• Add support for running jobs

0.1.8

• Add support for not equal

0.1.7

• Support multiple functions specified using `with/2`
• Add support for greater than or less than operators
• Add state storage option for output events or lookup

0.1.7

• Add job execution and timings
• Add state storage option

0.1.7

• Add job execution and timings
• Add state storage option

0.1.6

• Add notfound event matching

0.1.5

• Rewrite to make highly crash resilient
  • per module supervision
  • statistics data recovery
• Add wildcard event matching
• Add reset counters