When calling the slicer API, duplicate keys are not filtered which causes the highcharts line chart to break

Any dimension is accepted right now. The result ignores the dimension labels and draws the data points in order.

Example:

my_slicer.highcharts.line_chart(
	metrics=[['metric1']], 
	dimensions=['date'], 
	dimension_filters=[
		RangeFilter('date', start=date.today() - timedelta(days=3), stop=date.today())
	],
	operations=[L1Loss('metric1', 'metric2')],
	)


Exception:
Traceback (most recent call last):
File "<input>", line 7, in <module>
File "/Users/user/.venv/dashmore/lib/python3.4/site-packages/fireant/slicer/managers.py", line 420, in _get_and_transform_data
return tx.transform(dataframe, display_schema)
File "/Users/user/.venv/dashmore/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 78, in transform
series = self._make_series(dataframe, dim_ordinal, display_schema)
File "/Users/user/.venv/dashmore/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 110, in _make_series
for i, (idx, item) in enumerate(dataframe.iteritems())]
File "/Users/user/.venv/dashmore/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 110, in <listcomp>
for i, (idx, item) in enumerate(dataframe.iteritems())]
File "/Users/user/.venv/dashmore/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 115, in _make_series_item
'name': self._format_label(idx, dim_ordinal, display_schema, reference),
File "/Users/user/.venv/dashmore/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 161, in _format_label
metric_label = metric['label']
TypeError: string indices must be integers

Add validation for requests to the transformer managers so that exceptions are thrown for invalid requests before the query is executed:

Line Charts:
Require 1 continuous dimension as the first dimension

Bar/Column Charts:
Require maximum 2 dimensions+1 metric or 1 dimensions+n metrics

This should apply to both matplotlib and highcharts.

Fireant does not allow datatables sorting on metric columns that have a suffix or a prefix. A 'sort' key can be added specifying that sorting should be carried out on the raw value (without the prefix/suffix) which would fix this.

The pandas dataframe transformer performs a lot of steps which are common to all transformers such as setting the the display labels for metrics, dimensions, and dimension values. Make this happen automatically in the slicer manager to reduce the complexity in all of the transformers.

When using any type of reference in the WidgetGroup, an unexpected error occurs when slicing the data frame for each widget

Finish adding transformers for use in jupyter notebook. Use pandas for tables and matplotlib for charts.

Data tables results are transformed with np.int64 values in the results which cannot be serialized to JSON. Please cast these to python int.

import json

res = my_slicer.manager.column_index_table(
    metrics=['clicks', 'impressions'],
    dimensions=['date', 'device_type'],
    dimension_filters=[RangeFilter('date', date.today() - timedelta(days=14), date.today())])

print(res)
print(json.dumps(
    res
))

Example call:
res = slicer.highcharts.line_chart( metrics=['clicks'], dimensions=['date'], dimension_filters=[RangeFilter('date', date.today() - timedelta(days=14), date.today())], operations=[Totals('date')] )

Traceback (most recent call last): operations=[Totals('date')] File "/lib/python3.4/site-packages/fireant/slicer/managers.py", line 279, in _get_and_transform_data return tx.transform(df, display_schema) File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 47, in transform series = self._make_series(data_frame, dim_ordinal, display_schema) File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 90, in _make_series for idx, item in data_frame.iteritems()] File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 90, in <listcomp> for idx, item in data_frame.iteritems()] File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 95, in _make_series_item 'data': self._format_data(item), File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 168, in _format_data for key, value in column.iteritems() File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 169, in <listcomp> if not np.isnan(value)] File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 173, in _format_point return (_format_data_point(x), _format_data_point(y)) File "/lib/python3.4/site-packages/fireant/slicer/transformers/highcharts.py", line 13, in _format_data_point if np.isnan(value): TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'

Create the first post-operations for L1 and L2 Loss. These should replace the TODO in the slicer manager data function.

The dashboard WidgetGroupManager (fireant.dashboards.managers) raises exceptions due to metrics now being a 2 dimensional list. This error will have been introduced in the commit containing the new multi-axis feature for metrics.

Add connection to the Slicer schema so that each slicer can specify its own if necessary

Using a line chart with a widget group throws a TransformationException requiring a continuous dimension when the first dimension is implicit

Add operations for cumulative sum and cumulative average.

The ContainsFilter is raising an exception when it is used.

Example call:
res = searchads_slicer.highcharts.line_chart(
metrics=['clicks'],
dimensions=['date'],
dimension_filters=[RangeFilter('date', date.today() - timedelta(days=14), date.today()),
ContainsFilter('device_type', ['c'])],
)

print(res)

print(json.dumps(res))```

Error:

Traceback (most recent call last): File "", line 205, in <module> ContainsFilter('device_type', ['c', 't'])], File "/site-packages/fireant/slicer/managers.py", line 280, in _get_and_transform_data references=references, operations=operations) File "/site-packages/fireant/slicer/managers.py", line 69, in data references=references, operations=operations) File "site-packages/fireant/slicer/managers.py", line 103, in query_schema self._default_dimension_definition), File "site-packages/fireant/slicer/managers.py", line 205, in _filters_schema filters_schema.append(f.schemas(definition)) File "site-packages/fireant/slicer/filters.py", line 58, in schemas return element.isin(self.values) AttributeError: 'Coalesce' object has no attribute 'isin'

Exception raised in prevalidate request due to the interval being included in the keys.

The topic "FireAnt - Analytics and Reporting" appears 3 times in the documentation index (i.e. https://fireant.readthedocs.io/en/latest/index.html). Also, the Slicer page (i.e. https://fireant.readthedocs.io/en/latest/2_slicer.html) have some content that is shown 3 times as well. Finally, the latter page has "start-after: _appendix_start: :end-before: _appendix_end:" displayed, which is reStructuredText syntax that is probably not being parsed correctly.

pandas 0.20.1 results in test failures

If a user creates a unique dimension where the definition and display name both use a non-ID column e.g. a name, this will break the data tables. This is because the name often contains spaces or special characters and this breaks the data tables orthogonal data implementation.

One way around this would be to always hash the values used as keys in the datatables configuration that FireAnt creates. For example, hashing using hashlib and returning the hexdigest of the hashed value. This would ensure that no weird characters or spaces would appear in the keys.

Unique dimensions are sorted based on the ID field rather than the display field in the data tables result.

Please switch the render types to {'_': 'display', 'type': 'value'} so that the display values are used primarily.

Datetime dimensions used for column/bar charts are displayed with an ordinal integer label. Please change this to an ISO string of the date

A completely null value for a metric across all dimensions results in a NaN in the pandas dataframe which is then filtered out by the transformers.

All requests using totals results in an exception

File "/Users/llira/.virtualenvs/dashmore/lib/python3.4/site-packages/fireant/slicer/managers.py", line 395, in _get_and_transform_data
   display_schema = self.manager.display_schema(metrics, dimensions, references, operations)
 File "/Users/llira/.virtualenvs/dashmore/lib/python3.4/site-packages/fireant/slicer/managers.py", line 153, in display_schema
   'metrics': self._display_metrics(metrics, operations),
 File "/Users/llira/.virtualenvs/dashmore/lib/python3.4/site-packages/fireant/slicer/managers.py", line 324, in _display_metrics
   metric_key = getattr(operation, 'metric_key')
AttributeError: 'Totals' object has no attribute 'metric_key'

When rendering a dashboard table, if a Suffix has been set in the Slicer e.g. a Euro sign, this should not appear in the table values for columns showing a reference delta percentage on a metric.

I have a slicer that returns an empty dataframe (as expected) when called with

my_slicer.manager.data()

but when called with

my_slicer.notebooks.row_index_table()

it throws an Exception as follows:

(some parts of the stacktrace removed)

/Users/aweller/.virtualenvs/antfarm-strategies34/lib/python3.4/site-packages/fireant/slicer/managers.py in _get_and_transform_data(self, tx, metrics, dimensions, metric_filters, dimension_filters, references, operations)
    464         display_schema = self.manager.display_schema(metrics, dimensions, references, operations)
    465 
--> 466         return tx.transform(dataframe, display_schema)

/Users/aweller/.virtualenvs/antfarm-strategies34/lib/python3.4/site-packages/fireant/slicer/transformers/notebooks.py in transform(self, dataframe, display_schema)
     15 class PandasRowIndexTransformer(Transformer):
     16     def transform(self, dataframe, display_schema):
---> 17         dataframe = self._set_display_options(dataframe, display_schema)
     18         if display_schema['dimensions']:
     19             dataframe = self._set_dimension_labels(dataframe, display_schema['dimensions'])

/Users/aweller/.virtualenvs/antfarm-strategies34/lib/python3.4/site-packages/fireant/slicer/transformers/notebooks.py in _set_display_options(self, dataframe, display_schema)
     37 
     38                 if isinstance(dataframe.index, pd.MultiIndex):
---> 39                     dataframe.index.set_levels(display_values, key, inplace=True)
     40 
     41                 else:

/Users/aweller/.virtualenvs/antfarm-strategies34/lib/python3.4/site-packages/pandas/indexes/multi.py in set_levels(self, levels, level, inplace, verify_integrity)
    236             if not is_list_like(levels):
    237                 raise TypeError("Levels must be list-like")
--> 238             if is_list_like(levels[0]):
    239                 raise TypeError("Levels must be list-like")
    240             level = [level]

IndexError: list index out of range

Refactor references so they can be extended externally. Use an interval property which is passed to the query builder.

Using the references feature (http://fireant.readthedocs.io/en/latest/2_slicer.html#comparing-data-to-previous-values) throws a NotImplementedError on the interval function. I understand that I need to override the interval function in my subclass of Database, but what exactly is the function supposed to do?

Using the term label is confusing when a dimension has a label field for a human readable definition of the dimension, but also a label_ field for filtering by the label e.g. a keyword name. Maybe this label_field could be called display_field.

An exception is being raised

Collecting fireant[vertica] from git+https://github.com/kayak/fireant.git@master#egg=fireant[vertica] (from -r bandit/requirements.txt (line 51))
  Cloning https://github.com/kayak/fireant.git (to master) to /private/var/folders/k4/3gh_7drn1t78lh8hzlz8w6t08hdy18/T/pip-build-8g2fcnjb/fireant
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/k4/3gh_7drn1t78lh8hzlz8w6t08hdy18/T/pip-build-8g2fcnjb/fireant/setup.py", line 36, in <module>
        long_description=readme(),
      File "/private/var/folders/k4/3gh_7drn1t78lh8hzlz8w6t08hdy18/T/pip-build-8g2fcnjb/fireant/setup.py", line 12, in readme
        return f.read()
      File "/Users/mike/bin/../lib/python3.4/encodings/ascii.py", line 26, in decode
        return codecs.ascii_decode(input, self.errors)[0]
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 7399: ordinal not in range(128)```

As per the example at http://fireant.readthedocs.io/en/latest/2_slicer.html, running the following code is returning three columns instead of the expected seven.

A column-indexed table will contain only one index column and display a metrics column for each combination of subsequent dimensions. For example with the same parameters as above, the result will include seven columns: Day, Clicks (Desktop), Clicks (Tablet), Clicks (Mobile) and Conversions (Desktop), Conversions (Tablet), and Conversions (Mobile).

slicer.manager.column_index_table( metrics=['clicks', 'conversions'], dimensions=['date', 'device_type'] )

This is however returning three columns: Day, Clicks (clicks) and Conversions (conversions).

Null values are appearing with the value 'Total' in the datatables row table output if coalesce is not used.

When more than one dimension is specified, column index tables are breaking due to an extra column being added to the list of columns without a dimension name if null values occur in the returned data. This causes data tables to throw an exception and fail to render.

Currently it is required that the Dimension used by a reference is selected in order to use the reference. Please change this so that the reference can work with a dimension even if it is not in the query.

Returning data when using the Rollup operation for any transformer type e.g. column_index_table or line_chart results in an error.

Example call:
print(json.dumps(slicer.manager.line_chart( metrics=['clicks'], dimensions=['date'], dimension_filters=[RangeFilter('date', date.today() - timedelta(days=60), date.today())], operations=[Rollup(['date'])] )))

Error message:
TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Pypika has some bug fixes which need to be pulled in

When using a filter that requires a metric or dimension using a join, the joins are not added to the query.

Add logs to fireant. Log the query as debug, log requests as info.

Specifying column_name.label in a filter definition makes sense when the key is a column name, however as the key can be any unique value e.g. a hyperlinked URL or a unique ID, calling .label on this to specify that you want to filter on a label would be strange.

Example: dimension_filters=[WildcardFilter('account.label', 'nam%')]

When applying a ContainsFilter to a UniqueDimension, an exception is thrown 'list' object has no attribute 'isin'. The list id_fields needs to be handled.

When running code very similar to the example for rendering a row_index_table and attempting to output this as JSON data, an error is raised:

...raise TypeError(repr(o) + " is not JSON serializable") TypeError: Timestamp('2016-06-12 00:00:00') is not JSON serializable

Code causing the error:
slicer.manager.row_index_table( metrics=['clicks', 'revenue'], dimensions=['date', 'device_type'], dimension_filters=[RangeFilter('date', date.today() - timedelta(days=60), date.today())] )

Example Python dictionary returned from the above FireAnt statement (Note the Pandas Timestamp data type which cannot be converted to JSON):

{'Clicks': 123, 'DT_RowId': 'row_171', 'Date': Timestamp('2016-08-02 00:00:00'), 'Device Type': 'Tablet', 'Revenue': 123}, {'Clicks': 123, 'DT_RowId': 'row_172', 'Date': Timestamp('2016-08-03 00:00:00'), 'Device Type': 'Tablet', 'Revenue': 123},

Column charts are returning data points with NaN values if null values exist.

JSON returned for a column-indexed table is missing an entry for the totals although it is included in the list of columns.

In fireant/slicer/transformers/highcharts.py an exception is thrown on slicer.dimensions[dimensions[0]] when there are no dimensions on a slicer.

        from fireant.slicer import ContinuousDimension
        dimension0 = slicer.dimensions[dimensions[0]] ...
        if not dimensions or not isinstance(dimension0, ContinuousDimension):
            raise TransformationException('Highcharts line charts require a continuous dimension as the first '
                                          'dimension.  Please add a continuous dimension from your Slicer to '
                                          'your request.')

In highcharts line charts, if a multiple dimensions are used, each line for the same metric is the same color. This should be changed to be a different color for each metric/dimension combination.

The line for Clicks and Clicks(WOW) for example should be the same colour on a line chart.

Add a setting for the maximum number of columns on both pandas and datatables column-indexed tables to prevent extreme amounts of columns being rendered.

The default should be 24 columns.

Categorical dimensions are appearing with a label of None in column indexed tables, as if a display label has not been defined by the user, None is being set. This should show the raw db value instead of a prettified user defined value if no user defined value is provided.

The JSON returned when using the row_index_table or column_index_table managers returns data in a label: value format.

This causes issues with the datatables.net API as space characters may appear in the column name e.g. 'exchange rate'. When mapping the columns to a table in datatables, having spaces in the column name causes the data table to raise an exception as can be seen from the fizz buzz column below.

Example JSON returned from FireAnt:

{ "draw": 1, "data": [ { "foo": 12.1111 "bar": "1.1", "fizz buzz": 123, "DT_RowId": "row_0" }, { "foo": 12.1111 "bar": "1.1", "fizz buzz": 123, "DT_RowId": "row_1" }, ], "recordsFiltered": 2, "recordsTotal": 2 }

Example of column indexes to the columns found in the data:
$(#test).DataTable({ displayLength: 25, search: false, lengthMenu: [[10, 25, 50, 100, -1], [10, 25, 50, 100, 'All']], orderCellsTop: true, filtering: true, destroy: true, data: data, order: [0, 'desc'], columns: [ {data: 'foo'}, {data: 'bar'}, {data: 'fizz buzz'} ], ...

The exception returned is jquery.js:8264 Uncaught (in promise) TypeError: Cannot convert a Symbol value to a string(…).

There are a few solutions to how we could overcome this.

1. Ensure each JSON key for a column value has no spaces e.g. "fizz_buzz": 5. Users will have to work out a human friendly name for each column's header programatically.
2. Ensure each JSON key for a column value has no spaces e.g. "fizz_buzz": 5, but a label field in the nested data that a user can use to dynamically create a header column with a friendly name. e.g. "fizz_buzz": {"label": "Fizz Buzz", "value": 5}. Example at: https://datatables.net/examples/ajax/orthogonal-data.html
3. Return data using an array of arrays instead of an array of objects. Then the user can either work out what header should map to an array index or maybe FireAnt could pass an array which defines a label for each column, which could then be used by the user to set a human friendly header. Example at: https://datatables.net/examples/ajax/simple.html.

Example

import json
   res = _slicer.highcharts.line_chart(
       metrics=['clicks'],
       dimensions=['date'],
       dimension_filters=[RangeFilter('date', date.today() - timedelta(days=14), date.today()),
                          EqualityFilter('column', EqualityOperator.eq, 'value')],
   )
   print(res)
   print(json.dumps(res))

Result:

{'xAxis': {'type': 'linear'}, 'chart': {'type': 'line'}, 'title': {'text': None}, 'yAxis': [{'title': None}], 'tooltip': {'shared': True}, 'series': [{'yAxis': 0, 'data': [], 'name': 'clicks', 'dashStyle': 'Solid'}]} {"xAxis": {"type": "linear"}, "chart": {"type": "line"}, "title": {"text": null}, "yAxis": [{"title": null}], "tooltip": {"shared": true}, "series": [{"yAxis": 0, "data": [], "name": "clicks", "dashStyle": "Solid"}]}