advanced querying about core HOT 13 CLOSED

Sorting implemented client side, but I think we need to determine filtering on the server side and pass back only matches that are significant.

from core.

We can achieve this by implementing a server-side API that our frontend queries to get back filters/sorts.

from core.

Our "frontend" can be integration of this API with datatable's support for server-side operations.

from core.

Implement custom/advanced querying

from core.

My opinion of this is that it's not going to be used a lot, and will require a lot of indexes and work to build an interface. Albert still wants to give it a shot, so reassigning.

from core.

There are two parts to this:

1. Getting the data. We can either construct some complicated mongo query for this, or use something like elastic search, which albert wants to play with.

2. Building a workable UI for users to construct queries.

from core.

Advanced querying also includes aggregating the results for these kinds of queries.

There are two types of queries, ones that are predefined by YASP and we build something with the data, like a histogram, or nick's ward map, and ones that users build using some custom UI.

A key difference between automatic and manual queries is that automatic queries can occur on the server side prior to rendering data back to the user. Custom queries require hitting an API endpoint and sending back results.

Examples of automatic queries (We already build histograms of match duration and GPM)
select user, gpm, gets back array of gpms, build a histogram with it. Could also report numerical summaries.
select user, match duration, same as above
select user, runes, gets back array of hashes of counts by rune type.
select user, Kills/Deaths/Assists, get back array of hashes of counts for each
consumables avg/cumulative
item timings, for each item, hero?
HD/TD/HH (histograms)
Chat (ggs called/messages, Swearing/profanity analysis)
Grouping of heroes played (by valve groupings/primary attribute)
@nickhh 's idea, select user, obs, get back array of hashes of x,y,value, iterate through all of them, sum the totals, build a heatmap.

Examples of custom queries:
User selects user, spectre, radiance, get back array of radiance timings.

After the result of an advanced query, we want two things back:
returning the full array of values (with this we can display a table, or build a histogram)
returning the min/max/avg/sum (with this we can display a numerical summary of that data set)
Optionally, we also return some metric such as win/loss so we can get a winrate for each particular query.

Part 1 is building the functions to return these desired results for some given query.
Part 2 involves building a UI to allow users to build custom queries, then use the same functions for data.

Sorting: As long as the result set size is under 16MB (MongoDB's maximum) we can sort/filter on JS side (which is better since it doesn't require an index anyway).
Filtering: detect no stats recorded (algorithmically), significant game modes only

from core.

Taking a stab at this :)

from core.

so I think this is actually two separate things:

1. An advanced querying function that accepts conditions/return values, and filters an array of matches accordingly. It can also compute a winrate if a user was defined.

2. An aggregator function that accepts an array of matches, and returns aggregated data across specified fields.

from core.

I think I've done pretty much all I want to do with this, by adding a hero filter on the player trends page. The result is basically aggregattion by player, with an additional optional filter by hero. If we add too many more filters, the dataset will be too small to draw any meaningful conclusion.

If we want to provide some kind of aggregation api that works across players, I think we'll need to build a second api endpoint. We can't use the current /api/matches endpoint as that has to be limited heavily (in terms of n) in order to prevent abuse, as it can return full matches (which are 200kb each).

from core.

guess I will give this another shot. Current plan of attack:
Enhance the /api/matches endpoint:
Restrict projection per match to a very small set (to prevent bandwidth abuse)
Set a limit on how many matches we will aggregate (max result set size)
Set a limit on results returned at once to prevent bandwidth abuse
Do aggregation on the results (Since aggData is relatively small)?
Or we could just return winrate, count by iterating through the result set

Player matches page can use this endpoint to populate matches?

Possibly trends can be adapted to use this as well.

The problem with using datatables to interface with the api is that the query string suddenly gets a lot harder to build. Maybe we can add jQuery helpers to construct the query and use it for both matches tab (lists matches, reports winrate fitting the advanced query conditions) and trends tabs (aggregates data fitting the advanced query conditions)

from core.

I'm happy with the basic implemenation we have now. We can add more filters incrementally if users request them.

Over to Albert for styling.

from core.

framework is there for additional options to be added.

from core.